Subject: [Tsvwg] (SCTP) Congestion window calculation and bundling

From: Stephane Chazelas <[email-masked]>
Date: Fri, 17 Nov 2006 16:37:24 +0000

Hiya,

I can't imagine this has never been discussed in the past, but I
couldn't find archives of such past discussions, sorry if that's
a FAQ.

For the congestion window calculation algorithm, we measure how
much data we have in flight (how much we sent that has not been
acknowleged).

Now for the "how much data", we take into account only the
number of bytes of transported data, not the IP header, not the
transport header nor the SCTP data chunk headers.

For instance, the initial congestion window as defined by the
newer SCTP RFCs is likely to be something over 4000 bytes (or 4
MTUs or 2 MTUs) and is going to be the minimum as well.

That means that a SCTP stack can't send more than 4000 bytes of
data without waiting for an acknowledgement.

But there are many ways to send 4000 bytes of data with SCTP, as
SCTP is not byte oriented but stream of messages oriented.

For instance, you can send it as one 4000 byte message that
will be fragmented into three DATA chunks in 3 packets if your
MTU is 1500 for instance, or 4000 one byte messages, that will
generate 4000 DATA chunks.

Those 4000 DATA chunks will have either to be sent each one in a
separate packet (iphdr(=20) + sctphdr(=16) +datahdr(=16) +
data(=1) + padding(=3) == 56 bytes for a single byte of data),
or if the stack supports bundling (and the MTU is still 1500)
there will be about 70 DATA chunk in each packet so about 60
packets only.

To summarise, on startup, if I read the RFC correctly, a stack
can load the network with either of
- 3 fullsize (almost) packets (~4000 bytes).
- ~60 fullsize packets (~90000 bytes)
- 4000 56 byte packets (224000 bytes)

and anything ranging inbetween without having to wait for an
acknowledgement.

And it's the same for TCP if bundling is disabled and the user
sends one byte at a time.

When calculating the amount of data we have in flight, why don't
we account the amount of data carried by the network (the IP
packet length) rather than that the amount of data transported
(by the transport level)?

Surely 60 fullsize packets transporting 4000 one byte data
chunks (4000 byte worth of data) will congest a network the same
way as would 60 fullsize packets carrying 60 1500 bytes data
chunks (90000 byte worth of data).

Best regards,
Stephane

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 17 Nov 2006 18:54:18 +0100

Hi,

see my comments in-line.

Best regards
Michael

> MTUs or 2 MTUs) and is going to be the minimum as well.
I think the intention is to limit the number of messages...
> data(=1) + padding(=3) == 56 bytes for a single byte of data),
sctphdr=12 bytes.
> - 4000 56 byte packets (224000 bytes)
This is not true if you take Nagle into account. This will slowdown
things. Also if you chose the last possibility, you will very
likely see packet drops, which is not what you want.
> (by the transport level)?
Well, you might not know the IP stuff in the transport layer.
But I think it would be best to take the SCTP DATA chunk
headers also into account. This still does not take a possible
bundled SACK into account.
> chunks (90000 byte worth of data).
Sure. That is why I think the above is the best. We did some simulations
and found that the throughput is not affected that much, but the
load offered to the network. This means a lot of packets were dropped.

---

From: Stephane Chazelas <[email-masked]>
Date: Fri, 17 Nov 2006 18:08:44 +0000

On Fri, Nov 17, 2006 at 06:54:18PM +0100, Michael Tuexen wrote:
[...]
> load offered to the network. This means a lot of packets were dropped.
[...]

Thanks Michael for your comments.

I agree with you about accounting for the DATA chunk headers.
That would make SCTP behave as TCP.

Should we suggest the SCTP RFC be modified?

Cheers,
Stephane

---

From: Michael Tuexen <[email-masked]>
Date: Fri, 17 Nov 2006 19:50:43 +0100

Hi Stephane,

my personal opinion is that this might a something to put
into a BCP. This can discuss the effects of doing is this or
that way. There are implementations out there which do it
this or that way.

Best regards
Michael

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 17 Nov 2006 21:50:29 -0700

Stephane,

Stephane Chazelas wrote:                  (Fri, 17 Nov 2006 16:37:24)
> - 4000 56 byte packets (224000 bytes)

The Max.Burst parameter limits the sender to 4 packets.

Also, understand that many implementations will only send a 1
byte data chunk if a one byte record was written by the
application (I don't know of any such applications or even that
an application doing this could expect to be useful).  Most
applications fill data chunks quite a bit larger than the data
chunk header (otherwise the ensuing overhead would reduce
throughput to a trickle and would only be useful for low packet
volume applications), dropping your ratio from 60:1 to less than
2:1 in practice.

Also, many implementations of SCTP (following TCP) add the
buffering overhead of each write to the send buffer.

So assuming that there exists an application that will
pointlessly send 1 byte records, can manage to turn around 4000
sendmsg() calls fast enough, will not be limited by the send
buffer size, it will still be limited by Max.Burst and if a
packet drop results from sending the resulting 4 full size
packets, RTO will be doubled, further backing off the sending
rate.

As to the 4000 case, proper use of Nagle/Minshall applied to
SCTP would reduce this to the 60 full size packets case (well,
61); but this is no different from the situation for TCP.  TCP
can easily send 4000, 1-byte packets with "Nagle" disabled as
well.  In this case, where SCTP throttles to Max.Burst, TCP can
blast away.

The only difference from TCP that I see there is that Nagle is
required to be in effect by default for TCP, whereas it is
completely optional for an SCTP implementation.  It is my
experience that Nagle/Minshall applied to SCTP _always_
increases network performance for real-world applications.

Aside from Nagle not being required for SCTP, I don't see a
problem.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Sat, 18 Nov 2006 10:11:58 +0000

> The Max.Burst parameter limits the sender to 4 packets.

Hi Brian,

isn't maxburst only for heartbeats? I did observe the phenomena
of 15 fullsize packets being sent in a row, SACKs with rwnd
reduced to zero (because the receiver charges a big overhead per
chunk) being ignored. That would be a bug in that stack then?

> an application doing this could expect to be useful).

telnet comes to mind, but then the rate would be low. SCTP was
designed for sigtran to carry bounded messages. Applications or
upper layers may send small messages.

> volume applications), dropping your ratio from 60:1 to less than

You don't know all the past and future usages of SCTP.
Throughput is not necessarily the primary objective.

> buffering overhead of each write to the send buffer.

I'm not too sure what you mean by that.

> packets, RTO will be doubled, further backing off the sending

But the cwnd will not be reduced any lower than 4000 bytes (or 2
MTUs or 4 MTUs).

> Aside from Nagle not being required for SCTP, I don't see a

My concern was also about consistency. If the purpose is to
avoid congestion, it seems to make more sense to me to account
what really has an impact on the congestion (the amount of data
carried by the network).

Cheers,
Stephane

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sat, 18 Nov 2006 04:28:59 -0700

Stephane,

Stephane Chazelas wrote:          (Sat, 18 Nov 2006 10:11:58)
> chunk) being ignored. That would be a bug in that stack then?

RFC 4460 section 2.14.

Although the parameter Max.Burst is defined as 4 in
draft-ietf-tsvwg-2960bis-02.txt the text change from RFC 4460
section 2.14 does not seem to appear anywhere in 2960bis.

Randall: was that an oversight?  Or was RFC 4460 section 2.14
intentionally not included in 2960bis?  (If so, Max.Burst
parameter should be removed from the recommended parameters
seciton...)

> upper layers may send small messages.

Telnet uses Nagle to acheive better throughput.  Also, telnet is
not a single byte record application: it is a byte stream
application.

> Throughput is not necessarily the primary objective.

I think that the past (and envisioned future) IETF usages of SCTP
are a matter of public record.

> I'm not too sure what you mean by that.

Send and receive buffers are used to limit local system resource
usage.  A system that allows a user to allocated 1M of socket
buffers (with one byte of data) with a 1M send buffer is not
doing its job.  Most implementations subtract the socket buffer
overhead from the available send buffer.  On most systems the
buffer overhead is fairly large and you could easily not be able
to fit 1000 1 byte buffers into a 32k or 64k send buffer.

> But the cwnd will not be reduced any lower than 4000 bytes (or 2

The principle of both TCP and SCTP congestion windows is that
during slow start, or once the minimum cwnd is in effect, that
the sender still backs off on its retransmission rate by
doubling RTO.  The effect is a halving of the average sending
rate.  When all senders do this, the effect is a halving of the
traffic offered to a congested network.  At the minium cwnd it
is not the absolute number of packets offerred that provides
stability but the backing off of the RTO.

See Sally's recent draft-floyd-cc-alt-00.txt

> what really has an impact on the congestion (the amount of data

Sliding congestion window avoidance mechanisms do not depend
upon the absolute value of congestion window at any given point,
but rely upon the dynamic of the window instead.  The
consistency necessary is that the implementation count bytes for
congestion window calculations on each cycle the same.

When in slow start, if there exists such an application will
send 4000 1 byte records initially that experience a congested
network, it will send at half the previous rate on the next go
around (assuming no Max.Burst).  During normal operation, if
the application wants to waste the portion of bandwidth that it
has available to it by sending 1 byte data chunks in individual
packets, it is welcome to do so.  The congestion algorithm will
still not allow it to take up more than its share of network
capacity (data chunk headers and all).

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 18 Nov 2006 13:12:45 +0100

Hi Stephane,

Max.Burst applies also to DATA.

Best regards
Michael

---

From: Stephane Chazelas <[email-masked]>
Date: Mon, 20 Nov 2006 17:30:54 +0000

> parameter should be removed from the recommended parameters

Thanks Brian,

I can see that section now.

But the suggested implementation is to reduce the cwnd to
flightsize+4*MTUs if less than cwnd, which allows a stack to
send 4*MTUs worth of *data* (not IP traffic), so allows for the
4000 or 60 packets we were talking about, so that it's bringing
water to my water mill as the French say.

> not a single byte record application: it is a byte stream

It is. But if telnet were to use SCTP with the socket API
without further change, it would send data from the keyboard as
it comes, so would generate 1 byte chunks. Generally, pressing
"A" would generate one DATA chunk and one data chunk containing
"A" would come back for display on the screen. If that latter
chunk is bundled with a SACK, that's fine. Otherwise, the
interactivity on links  with little delay would be reduced as
the next keypress would not be sent out immediately because of
the bundle timer/nagle. Is there a requirement for bundling a
SACK with a DATA if we've received DATA we've not acknowledged
yet? Or could a stack delay the sending of the SACK there? If
that were the case, a telnet implementation may chose to set the
nodelay option and that would make the problem worse as every
keypress would go in its own packet not only its own data chunk.

But still, imagine that scenario (extreme but to make the
point): a network is heavily congested, only two packets
can go at a time, and delay is huge (rtt 10 seconds).

A user presses and holds the "a" key. The key repeat timer makes it
send 50 characters per second. telnet does a while(nread =
read(0, buf...)) send(sock,  buf, nread). nread will always be 1.

The bundle timer is 400 ms for instance, MTU is 800 bytes, cwnd
4 MTUs (3200).

The SCTP stack will take 20 characters in, after 400ms send one
432 byte packet, inflight will be 20 (3180 left until the
congestion window is full).

After 10 seconds (at the time the first sack indicating only the
two first packets made their way (so far)), 500 bytes in 25
packets will have been sent, 23 of which will have been lost by
the network.

after the retransmission timeout, what prevents the stack to
send those 23 packets all at once again?

> are a matter of public record.

Maybe, but when you write a TCP/IP application, you don't ask
for the IETF approval. I think the SCTP specification should be
defined with that in mind. And as there's nothing preventing an
application to send messages of short and/or variable size, that
should be taken into account.

> to fit 1000 1 byte buffers into a 32k or 64k send buffer.

Also, I find it weird that you would not take into account the
data chunk overhead, relying on the implementation (the fact
that the cwnd will likely have an upper limit driven by the size
of the send buffer) in a document that describes the protocol.

AFAICS, the RFC doesn't say anything about implementation buffer
allocation and management

[...]
> > > increases network performance for real-world applications.

The other difference is that TCP, which doesn't have notions of
messages would not send 60 packets, but 3 (with a 1500 byte MTU)
to send 4000 bytes when nagle is on.

> congestion window calculations on each cycle the same.

That's the point. If an application sends 30 fullsize packets to
send 2000 bytes, and the next time sends 2 fullsize packets, a
congestion avoidance calculation that doesn't take the data
chunk header size into account won't make sense.

> around (assuming no Max.Burst).

My understanding is that it will resend those 60 fullsize
packets as it's just 4000 bytes which fit in cwnd.

> During normal operation, if
[...]

How?

I don't agree, the size of the packet is useful information to
whom uses it. Without it, uper layer such as HTTP could waste
even more space (like the Content-Length: 1234\r\n in HTTP).
It's information to the upper layer, so why not charge for it?

Best regards,
Stephane

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 20 Nov 2006 11:12:51 -0700

Stephane,

Stephane Chazelas wrote:                                          (Mon, 20 Nov 2006 17:30:54)
> water to my water mill as the French say.

It also says that the implementation can count packets to acheive the
desired effect (i.e. 4 packets) and this is also the most obvious and
easiest implementation, making your situation vaporize.

> without further change, it would send data from the keyboard as
-X--snip--X-

This is no different than the case of TCP with Nagle turned off:
individual one byte packets where neither the IP header nor TCP headers
are included in the cwnd calculation.  So what.  SCTP header and 1 data
chunk header vs. TCP header: big deal.

> should be taken into account.

Yes, such an application will experience extraordinarily poor throughput
performance, even poorer than for TCP.  So what?

> allocation and management

Because both are of only local significance.

> to send 4000 bytes when nagle is on.

Two things: 1) Nagle usually only half-fills packets, so for TCP there
will actually be 6 with Nagle on.  2) SCTP limits Max.Burst to 4, so
SCTP will send fewer packets than TCP.

> chunk header size into account won't make sense.

It makes perfect sense to me.  The sender of many 1 byte data chunks
will stand a higher probability that a packet is discarded (per byte of
data counted) and will suffer even poorer performance because of a
further reduced window.

You still don't understand the dynamic, do you?  Your poorly designed
telnet application send 120 packets to send 8000 bytes of data.  If the
network drops 1 packet in 120, its congestion window falls to 4000.  My
application sends 120 packets to send 180,000 bytes of data.  If the
same network drops 1 packet in 120, my congestion window falls to
90,000.  Counting the data chunk headers does not matter.  It is not the
absolute size of the window that counts, but the dynamics of the window.

> packets as it's just 4000 bytes which fit in cwnd.

I did not say that it would not send 60 packets, it sends them at half
the rate because RTO doubles.  This is sufficient.  Did you read Sally's
draft?

> > capacity (data chunk headers and all).

See example above.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Mon, 20 Nov 2006 19:13:16 +0000

> easiest implementation, making your situation vaporize.

My situation doesn't vaporize as I found at least one stack that
implements it the first way. And the fact that the RFC mentions
two approaches for the same thing that have two different effects
makes me think that the RFC (well the draft) authors missed that
particular point I'm raising.

And it's not obvious to me what the easiest implementation is.
At least Spider and Linux sctp implement it the first way. BSD
and OpenSS7 seem to implement it the second way.

> chunk header vs. TCP header: big deal.

The scenario I exposed was with nagle turned on, sorry for not
having made that clear.

> performance, even poorer than for TCP.  So what?

Let's take another example as you seem hard to convince :).

Joe has an application that reads sensor data from 123 sensors.
Those sensors sometimes send 1 sometimes 3, sometimes 5 bytes of
DATA at a time.

Joe used to use TCP to send the sensor information to Bill over
the internet. He used to send:

struct {
  u32 sensor_id;
  u32 data_length;
  char data[0];
}

records over a TCP connection.

He was not happy with that because he didn't need to guarantee
the sequencing of records inbetween sensors. So, he thought
about using SCTP.

With SCTP, it's easy, he can have one stream per sensor and he
doesn't need to worry about size as it's already handled by
SCTP. All he needs to do is sctp_send(sensortId, data, len).

Now, he discovers that performances are being a bit erratic,
why?

[...]
> SCTP will send fewer packets than TCP.

Not with stacks that implement the max.burst the way draft
describes it.

> absolute size of the window that counts, but the dynamics of the window.

I understand that point. But I do understand that if the minimum
cwnd was defined as 2 MTUs, then 4 MTUs, what was in mind the
equivalent of 2 or 4 fullsize packets. What I say is that even
if the delay inbetween two retransmission doubles every time,
your stack is going to resend 90kb even if the network can't
take more than 4 at a time. With TCP (and nagle), only 4 kb
would be sent.

In effect, the minimum MTU is not 4 MTUs but 60 MTUs then as
that's as much load SCTP can put on a network.

What's your argument against charging for the DATA chunk header?

regards,
Stephane

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 20 Nov 2006 12:42:29 -0700

Stephane,

Stephane Chazelas wrote:                          (Mon, 20 Nov 2006 19:13:16)
> and OpenSS7 seem to implement it the second way.

That would be because the requirement is to limit the number of packets
sent (to 4 which which is Max.Burst).  The first (optional) method does
not do that in your case.  I propose we strike the first option (since
it has not been included in 2960bis yet).  Then, vapor.

> The scenario I exposed was with nagle turned on, sorry for not

With Nagle turned off the application will be welcomed both in TCP
and SCTP to waste its available bandwidth.

> Now, he discovers that performances are being a bit erratic,

Because a byte stream protocol (TCP) is better suited to his micro
record application.  This will not be helped by counting data chunk
headers.

> Not with stacks that implement the max.burst the way draft

We'll strike the first option: then it will.

> I understand that point. But I do understand that if the minimum

And missed the previous one.

> take more than 4 at a time. With TCP (and nagle), only 4 kb

TCP will only send 4 kb per interval if it can get 4 kb per interval
through, otherwise it will drop to 2 kb per interval (4 kb over
twice the interval).  It continues until throughput offered to the
network is very small compared to the normal bandwidth handling
capacity.  So does SCTP.

> that's as much load SCTP can put on a network.

Traffic offered to the network is 4MTU/RTO and 60MTU/RTO.  It will
wind up that SCTP's RTO will be 15 times longer than TCPs.

> What's your argument against charging for the DATA chunk header?

Same reason as not charging the SCTP header, or the IP header, or the
Ethernet header, or the SACK, or the SHUTDOWN: all are packet overhead
and not payload.  SCTP simply has more overhead when abused, but it
does not break the congestion algorithm.  Why are you not counting the
TCP header, IP header, and Ethernet header when telnet sends 1 byte of
data in a packet with Nagle turned off?  (Because they do not matter to
a sliding window algorithm?)

Packets that can break the algorithm are packets with all overhead and
no payload.  This is one reason why data chunks with zero data are not
permitted, and why the number of SACKs sent is tied to packets containing
DATA.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Tue, 21 Nov 2006 08:11:34 +0000

> it has not been included in 2960bis yet).  Then, vapor.

If you limit the number of packets to 4, when are you going to
send the remaining ones? The draft doesn't seem to tell it.

I was wrong about BSD and OpenSS7. BSD seems to use the first
method, and the code that uses max_burst in OpenSS7 seems to be
commented out. But you may know that better than me, I was
looking at strsctp-0.9.2.5

Do we know why max_burst didn't make its way to RFC 2960bis?

> and SCTP to waste its available bandwidth.

Yes, but as far as congestion avoidance goes, you'd expect
applications that generate a heavy load to be using Nagle. That
may be why CA worked so far with TCP.

[...]
> record application.  This will not be helped by counting data chunk

Not agreed. Here the overhead for using SCTP is just less than 8 bytes
(TSN+SSN - TCP-SN/records-per-packet) per record. And if he
wanted to have the delivery of records per sensor independant,
he'd have to create one TCP connection per sensor, instead of
just one SCTP stream which would be more overhead.

And you may change the example above to instead of 1..3, 1..2048
with an average of 500 during the day and 3 at night.

> We'll strike the first option: then it will.

Then we need to give more details about how to implement the
second option.

[...]
> wind up that SCTP's RTO will be 15 times longer than TCPs.

With bursts of 60 fullsize packets that may cause havoc in
routers queues.

[...]

Again, the information in the DATA header can be considered
payload/useful information to the upper layer/application.

The congestion avoidance is to avoid network congestion. Sizes
are expressed in network units (MTUs), it'd make sense to me to
count TCP, IP and SCTP header, not ethernet obviously.

Charging for the DATA chunk header would make SCTP behave mostly
(the TSN would be accounted though) the same as TCP which we
know works mostly in cases where it matters most (for heavy
traffic with nagle/bundling).

Best regards,
Stephane

---

From: Stephane Chazelas <[email-masked]>
Date: Tue, 21 Nov 2006 10:45:11 +0000

Hi,

I wanted to summarise the /problems/ encountered so far:

- as the current RFC 2960bis (draft-ietf-tsvwg-2960bis-02.txt)
  is written, a sender is allowed to send (even upon association
  establishment) a burst of over 60 fullsize packets over a path
  with MTU of 1500 (it may even be worse with other values of
  MTU).

- the description about max burst in RFC 4460 doesn't appear in
  RFC 2960bis

- the first of the two methods described for implementing max
  burst in RFC 4460 doesn't prevent the problem above (the
  sending of 60 fullsize packets) even when the Max.Burst
  parameter is 4 as recommended.

- the second of those two methods doesn't specify, if the number
  of outgoing messages allowed to go at a time (how much time?)
  is limited, when we're allowed again to send the remaining
  packets.

This text is very vague to me and seems to make some assumptions
on the implementation:

   D) When the time comes for the sender to transmit new DATA chunks,
      the protocol parameter Max.Burst SHOULD be used to limit the
      number of packets sent.  The limit MAY be applied by adjusting
      cwnd as follows:

      if((flightsize + Max.Burst*MTU) < cwnd)
         cwnd = flightsize + Max.Burst*MTU

      Or it MAY be applied by strictly limiting the number of packets
      emitted by the output routine.

What if the output routine is called once for every packet. That
text kind of assumes the data chunks are queued/buffered in
some way. Is it linked to this note in the RFC:

   During times of congestion an SCTP implementation MAY still perform
   bundling even if the user has requested that SCTP not bundle.  The
   user's disabling of bundling only affects SCTP implementations that
                                             ~~~~~~~~~~~~~~~~~~~~~~~~~
   may delay a small period of time before transmission (to attempt to
   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   encourage bundling).  When the user layer disables bundling, this
   ~~~~~~~~~~~~~~~~~~~
   small delay is prohibited but not bundling that is performed during
   congestion or retransmission.

In which case, shouldn't the same note apply to max_burst? What
about retransmissions, is it OK to send bursts of retransmitted
packets?

regards,
Stephane

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 21 Nov 2006 12:00:51 +0100

Hi Stephane,

I think any text change described in RFC 4460 and missing in
the bis ID is an editorial mistake. But I'll leave it to Randall
for a final statement and I guess he'll fix it and put the text
in.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 21 Nov 2006 12:12:47 +0100

Hi Stephane,

comments in-line.

Best regards
Michael

>   with MTU of 1500 (it may even be worse with other values of
That is true. However, the sender does not get more bandwidth. Our
simulations show that you send a lot of packets which will be
dropped in the network which is bad. This can be avoided by taking
the DATA chunk header into account.
> - the description about max burst in RFC 4460 doesn't appear in
I'm sure that this is just a "bug". It will be fixed.
>   parameter is 4 as recommended.
They are similar, if
- the DATA chunk header is taken into account and
- Nagle is used.
>   is limited, when we're allowed again to send the remaining
It is the number of packets you can send when the sender will
put packets on the wire. Such events include
- the upper layer did a send, Nagle is on and there is no
   outstanding data.
- A T3 timeout
- A SACK arrived and new data can be sent.
> text kind of assumes the data chunks are queued/buffered in
But it says "When the time comes for the sender to transmit new DATA  
chunks"
and not "the output routine is called". So I do not see your point.

---

From: Stephane Chazelas <[email-masked]>
Date: Tue, 21 Nov 2006 13:27:44 +0000

On Tue, Nov 21, 2006 at 12:12:47PM +0100, Michael Tuexen wrote:
[...]
> the DATA chunk header into account.

Hi Michael,

Agreed.

[...]
> - the DATA chunk header is taken into account and

Agreed.

> It is the number of packets you can send when tFrom [email-masked] Tue Nov 21 08:28:12 2006
Date: Tue, 21 Nov 2006 13:27:44 +0000
From: Stephane Chazelas <[email-masked]>
To: Michael Tuexen <[email-masked]>
Subject: Re: [Tsvwg] (SCTP) Congestion window calculation and bundling
Cc: "Brian F. G. Bidulock" <[email-masked]>, tsvwg <[email-masked]>

On Tue, Nov 21, 2006 at 12:12:47PM +0100, Michael Tuexen wrote:
[...]
> the DATA chunk header into account.

Hi Michael,

Agreed.

[...]
> - the DATA chunk header is taken into account and

Agreed.

> - A SACK arrived and new data can be sent.

The RFC says "new data" (see below), so it shouldn't apply to T3
timeout, should it?

It says "new data chunks" but then talks about "number of
packets". What if the packets contain both new and retransmitted
chunks?

> But it says "When the time comes for the sender to transmit new DATA chunks"

It says the "number of packets emitted by the output routine",
if the output routine is called for every packet, then it won't
be applied. Ok, I may be picky there.

[...]

I got confused and go my ideas wrong here.

Even then a stack would wait to try and fit more data in one
packet, but when the current packet it is currently building is
full, it would output it. And that's a single packet.

So, that Max.Burst only applies when the user sends messages
more than 4 MTUs of size or when we're buffering data because
we've used up our congestion window and a SACK gives us back an
allowance of more than 4 MTUs.

In both those cases, the packets will be used up efficiently
(either because big packets are fragmented to maximise the MTU
or because buffering implies bundling).

>         cwnd = flightsize + Max.Burst*MTU

If we are buffering, that is cwnd+MTU > flightsize >= cwnd-???.
As the cwnd can increase by at most 1 MTU, the only way to have
an allowance for more than 4 MTUs is if a SACK acknowledges more
than 3 MTUs.  This algorithm may end up reducing the cwnd. Now,
as a stack is supposed to send a sack at least every other
packet, it may seldom happen, and when it does it may be an
indication there's something wrong in the network.

If we're just limiting the sending to 4 packets without altering
cwnd, then that must imply "until something happens". Now what
are the events that would allow us to send the remaining
packets in our buffer allowed by cwnd? SACK reception, PKTDROP
reception, more data from the user, CWR, ECN?

So that it does prevent bursts but only when we're pushing the
congestion window, in effect when we're flow controlling the
user, I think. If the congestion window is not fully utilised,
then the user can send bursts.

Sorry about the noise,
Stephane

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 21 Nov 2006 06:26:55 -0700

Michael,

Michael Tuexen wrote:                      (Tue, 21 Nov 2006 12:12:47)
> chunks" and not "the output routine is called". So I do not see your point.

When Nagle is off, a time comes to send new DATA chunks whenever the
output routine is called.

Nagle is, as you know, not required for SCTP and neither is bundling.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

)
	by megatron.ietf.org with esmtp (Exim 4.43) id 1GmVeQ-00040F-4W
	for [email-masked]; Tue, 21 Nov 2006 08:27:02 -0500
Date: Tue, 21 Nov 2006 06:26:55 -0700
From: "Brian F. G. Bidulock" <[email-masked]>
To: Michael Tuexen <[email-masked]>
Subject: Re: [Tsvwg] (SCTP) Congestion window calculation and bundling
Mail-Followup-To: Michael Tuexen <[email-masked]>,
	Stephane Chazelas <[email-masked]>, tsvwg <[email-masked]>
Organization: http://www.openss7.org/
Dsn-Notification-To: <[email-masked]>
Cc: Stephane Chazelas <[email-masked]>, tsvwg <[email-masked]>
Reply-To: [email-masked]

Michael,

Michael Tuexen wrote:                      (Tue, 21 Nov 2006 12:12:47)
> chunks" and not "the output routine is called". So I do not see your point.

When Nagle is off, a time comes to send new DATA chunks whenever the
output routine is called.

Nagle is, as you know, not required for SCTP and neither is bundling.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 21 Nov 2006 14:39:08 +0100

Brian,

comment in-line.

Best regards
Michael

> output routine is called.
Depends on what you call the output routine. But if nagle is of, rwnd  
and
cwnd allows sending, then you send a packet each time the user calls  
send()
or whatever the sending routine is...

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 21 Nov 2006 14:43:52 +0100

Hi Stephane,

see my comments in-line.

Best regards
Michael

> The RFC says "new data" (see below), so it shouldn't apply to T3
You are right.
> packets". What if the packets contain both new and retransmitted
retransmitted chunks would be in, because of T3, or SACK arrival.
> be applied. Ok, I may be picky there.
I think you mean by "output routine" something like send(). I was
thinking more in terms of sctp_output(), a kernel routine....

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 21 Nov 2006 06:53:52 -0700

Stephane,

Stephane Chazelas wrote:                            (Tue, 21 Nov 2006 08:11:34)
> looking at strsctp-0.9.2.5

It only impaired high end CMT performance at that place in the loop
and was largely ineffective (it simply re-enters from an outer loop).

> may be why CA worked so far with TCP.

Nagle is not required for SCTP and neither is bundling (at the sender).

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Tue, 21 Nov 2006 13:55:18 +0000

On Tue, Nov 21, 2006 at 01:43:52PM -0000, Michael Tuexen wrote:
[...]

> retransmitted chunks would be in, because of T3, or SACK arrival.

What I meant is that upon T3 or fast-retransmit, you wouldn't
limit the number of retransmitted packets, but "when the time
comes to transmit new DATA chunks", would you limit the new
packets, the old packets, the packets with old and new at the
same time?

> thinking more in terms of sctp_output(), a kernel routine....

Well, I did mean the routine that SCTP uses to deliver packets
to IP. It could be putnext in a STREAMS implementation, it could
be anything, you have to guess to know what is meant. I was a
bit picky as given some thought, the guess should not be too
hard.

-- 
Stephane

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 21 Nov 2006 07:01:46 -0700

Michael,

Michael Tuexen wrote:                                (Tue, 21 Nov 2006 14:43:52)
> > The RFC says "new data" (see below), so it shouldn't apply to T3

Don't think so...  When you are finished retransmitting outstanding packets
you are still allowed to send new data up to cwnd.  So, as a result of T3
expiry, it can come time once more to send new DATA packets.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Tue, 21 Nov 2006 14:24:52 +0000

On Tue, Nov 21, 2006 at 02:01:46PM -0000, Brian F. G. Bidulock wrote:
> Don't think so...  When you are finished retransmitting
> outstanding packets you are still allowed to send new data up
> to cwnd.  So, as a result of T3
[...]

Data that would have been held back because of Max.Burst, you
mean? A T3 timeout would shorten the cwnd though, so it's not
obvious you may be able to fit another packet though.

Am I the only one to find those areas of the specification very
vague?

I may help to know the initial purpose of Max.Burst and what
problem it is trying to solve.

RFC 4460 would suggest it is an adjustment on the cwnd
calculation algorithm. If the purpose is to limit the increase
(or even to decrease) of the cwnd when the window is not fully
utilised, then maybe a stack should go with the 

Which tells that if there's as much free room in the window as
4 MTUs, then don't increase, or even decrease the cwnd to keep
not more than 4 MTUs worth of free room in the window.

If the goal is to avoid bursts, that is avoid the sending of
many packets in a short period of time, then it's another
matter.

Regards,
Stephane

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 21 Nov 2006 18:47:44 +0100

Hi Stephane,

my understanding is that the max.burst stuff limits the
amount of data in case of you are not using the CWND...

Best regards
Michael

On Nov 21, 2006, at 3:24 PM, Stephane Chazelas wrote:

>> Michael Tuexen wrote:                                (Tue, 21 Nov  
>> 2006
>> 14:43:52)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 21 Nov 2006 14:13:59 -0700

Stephane,

Stephane Chazelas wrote:                                          (Tue, 21 Nov 2006 14:24:52)
> obvious you may be able to fit another packet though.

It is obvious to me.  Say your cwnd is 1000*MTU with nothing in
flight and the user writes 8*MTU.  According to max.burst you
send 4*MTU and wait.  No SACK arrives.  T3 times out.  cwnd
gets halved to 500*MTU.  You retransmit 4*MTU and can burst the
yet unsent 4*MTU.  Therefore, a T3 timeout is one of the events
that can cause a time to come to send new data.

> Am I the only one to find those areas of the specification very

I thought that there could be a better description too.

> problem it is trying to solve.

The problems are as stated in RFC 4460 2.14.  The major issue as
it was described to me is maintaining a large cwnd that is not
being utilized.  Take the example above with a few changes:
10000*MTU cwnd, nothing in flight, no data to send.  After several
seconds the user writes 10000*MTU.  All 10000*MTU get bursted to
the line at line rate.  If the network congested during those
several seconds idle period, you have just bursted 10000*MTU at
a congested network.  That number could easily be much larger.
If you are placing 1 byte in a data chunk and one data chunk in
a message, that is a lot of packets.

This is an extreme example, but, without max.burst, at any time
that cwnd is being underutilized, by  writing a sufficient amount
of data, the application can cause (cwnd - flightsize) to be
bursted.  max.burst limits the burst to 4*MTU (or 4 packets).

> not more than 4 MTUs worth of free room in the window.

I think that are reading this wrong.  As I understand it, "cwnd"
here is the cwnd that is used for the current cycle (calculating
how much can be send right now) and is not a permanent adjustment.
So it has no effect on how cwnd is calculated, just how it is
applied to the current transmission cycle.

The problem I had with the mechanism as described is that it
behaves differently under different user behavior.  There is a
difference if the user does this: write(fd, buf, 80000) or
does this: for (n=80000;n;n-=8) { write(fd, buf, n); buf+=8; }
and I do not think that there should be.  I do not think that
the user should be encouraged (by experiencing better throughput)
to do the later.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Wed, 22 Nov 2006 10:21:33 +0000

On Tue, Nov 21, 2006 at 09:13:59PM -0000, Brian F. G. Bidulock wrote:
[...]
> flight and the user writes 8*MTU.

Hi Brian,

If you've got nothing in flight, then the cwnd would be 4 MTUs
when using the first algorithm. That's with that in mind that I
thought it was not obvious that you could be in a position to
send data upon a T3 timeout.

And remember the cwnd is halved every rto if nothing is being
sent.

[...]
> bursted.  max.burst limits the burst to 4*MTU (or 4 packets).

Yes, I agree. When the cwnd is not utilised, it is decayed,
halved every rto. But, when it is underutilised, like in your
example if at some point the stack managed to have 10000 MTUs in
flight and then is now sending at a constant pace with 4 MTUs
in flight; without max.burst, you'd keep the same cwnd and the
network conditions might change without you noticing it and cwnd
may not reflect how much the network can take at the moment.

So, my understanding is that we need a mechanism for decaying
the cwnd not only when we're not sending but also when we're
under using the cwnd.

That's how I read the RFC 4460 comment.

If you limit the number of messages sent at a time to 4 MTUs
without altering cwnd, and only allow to send upon reception of
SACKs, that still allows you to increase your flight size very
quickly but that will smoothe that increase.

If cwnd is 10000 MTUs, and you have 100 MTUs in flight and the
effective cwnd (how much the network can take) is now 120 MTUs.
If your user is now giving you a lot more to send, you'll be
receiving 50 SACKs for the data you have in flight. If you're
sending at 4 MTUs per SACK, you'll be reaching the 120 flight
size after receiving about 10 SACKs and it will then take you 60
more SACKs before you detect your first packet loss (gap in a
SACK). By that time, you will have a flight size of about (100 +
10+60 * (4 - 2) == 240). It will take you a little longer to
reduce your cwnd to 5000 and then to 2500, 1200, 600, 300, 150,
75 MTUs, but in anycase it should be better than without the max
burst.

[...]
[...]

Well, that's not what I understood. And that's apparently not
what Linux, BSD and Spider SCTP implementers understood.

And a permanent adjustement fits well with the idea above of how
the cwnd can be decayed when under-utilised, that is don't allow
the cwnd to be too much above the current flightsize.

Another area that needs clarification, I'd say.

Cheers,
Stephane

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 23 Nov 2006 03:23:23 -0700

Stephane,

Stephane Chazelas wrote:                 (Wed, 22 Nov 2006 10:21:33)
> And remember the cwnd is halved every rto if nothing is being

That is only a should and not a must.  Nevertheless, recommended
RTO min is 1 second.  So, even if you did half cwnd every RTO
(busily running timers on idle destinations), and wait idle for
two seconds, the cwnd will still be 250*MTU.  When sending data
the same signal would be fed back from SACKs in two RTT and
it is typcial that RTT << 1 second.  So it does not affect the
example.

Don't use the first algorithm.  I said we should throw it out
anyway, because it does not limit the case you first brought up
in this thread (the number of packets).

> in flight; Well, that's not what I understood. And that's
> apparently not what Linux, BSD and Spider SCTP implementers
> understood.

It that was their interpretation then they were very foolish to
apply the first method (because the second method obviously does
not permanently affect cwnd in any way).  So we'll throw out the
first method, because it does not limit the number of packets
sent in to 4 in slow start.

> the cwnd to be too much above the current flightsize.

If cwnd were to be adjusted permanently, I would not characterize
it as decay, but instantaneous (and rather permanent) truncation.
But, we'll throw it out.

> Another area that needs clarification, I'd say.

No, just deletion.  The second method, strictly limiting the number
of packets sent, is quite clear.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Stephane Chazelas <[email-masked]>
Date: Thu, 23 Nov 2006 10:55:47 +0000

On Thu, Nov 23, 2006 at 10:23:23AM -0000, Brian F. G. Bidulock wrote:
[...]
[...]

Hi Brian,

yes, I agree with you that there's something wrong with that
approach. The cwnd would be reduced immediately to 4 MTUs after
one rtt when we stop sending which would conflict with the
normal (optional as you pointed out) decay that says the cwnd
should be halved every rto when we stop sending.

I'll look at the code of those stacks more carefully to see if
that's really what happens.

Maybe a third approach would be to do:

cwnd = (cwnd + flightsize) / 2

every rto (unless flightsize > cwnd).

regards,
Stephane

---

From: Stephane Chazelas <[email-masked]>
Date: Fri, 24 Nov 2006 11:01:40 +0000

On Thu, Nov 23, 2006 at 10:55:47AM -0000, Stephane Chazelas wrote:
[
  About the
	if((flightsize + Max.Burst*MTU) < cwnd)
	   cwnd = flightsize + Max.Burst*MTU
  approach at implementing max burst
]

[...]

Well it seems indeed that's how those stacks implement it, so
though I've not tested it, that would mean that the cwnd would
be reduced to 4 MTUs after one rtt instead of being halved every
RTO. For KAME however, you can select the behavior at compile
time, and for FreeBSD possibly at runtime.

Best regards,
Stephane

---

From: Randall Stewart <[email-masked]>
Date: Wed, 29 Nov 2006 11:19:05 -0500

> parameter should be removed from the recommended parameters

It was not intentional... and must have gotten missed
in the merge...

Let me go in and see if it is more "hidden text" within the
xml.. or if its just a cut and paste error.

R
-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Randall Stewart <[email-masked]>
Date: Wed, 29 Nov 2006 11:27:33 -0500

> Do we know why max_burst didn't make its way to RFC 2960bis?

BSD allows a compile option to do it the opposite way. aka counting.
The default is by adjusting the cwnd.

This whole topic was discussed in length on this list... Mark Allman
was the proponent of cwnd based adjustment.. Kacheong Poon was the
proponent of the counting method. Mark, if I remember right, was
concerned about having two control variables and their interaction.
Kacheong claimed that Solaris has done counting for years for TCP
with no harm.. thats what I remember of the debate.. without a search
of the archives :-)

As to why its not there.. I will go look.. it was obviously an
oversite/cut/paste error..

I first though, want to get through this long thread :-D

R

---

From: Randall Stewart <[email-masked]>
Date: Wed, 29 Nov 2006 11:33:08 -0500

> amount of data in case of you are not using the CWND...

My understanding is it is to try to limit micro-bursts
that can be generated by a stack...

R