Subject: Re: [Tsvwg] SCTP applicability statement

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 14:58:19 -0600

RJ,

RJ Atkinson wrote:                                             (Sun, 04 Nov 2001 15:36:56)
> At 14:31 04/11/01, Brian F. G. Bidulock wrote:
> >What do you mean by *internal*?   *Internal* to what?
> 
>         Internal to the "Cluster Operating System".
> 
> >>         This isn't necessarily rocket science.  However, just as the 
> >> methods for clustering vary widely from one implementation to another, 
> >> the internal methods for ensuring that everything works properly 
> >> in event of failover will need to vary.  Thus, such mechanisms
> >> do not seem ripe for standardisation in IETF.  If one wanted to
> >> standardise clustering methods, IEEE would seem like a more suitable
> >> venue as they have ongoing operating system standards efforts
> >> (and it looks like an OS problem).
> >
> >Are you suggesting that the SIGTRAN WG be dechartered and sent to IEEE?
> 
>         No.  I gather you either didn't understand or didn't read 
> what I wrote.  I'll try again with an effort to use simpler language.
> 
>         I'm suggesting that the details of how a cluster operating system 
> is built are not IETF work and that those details should be kept hidden 
> under the hood of the given cluster (i.e. not be visible on-the-wire).
> SIGTRAN isn't chartered to design a cluster operating system, nor
> is SIGTRAN chartered to modify transport protocols to solve clustering.
> Clustering computer systems is an OS issue not a protocol issue.

But this is what SIGTRAN *has* done by havinng multiple ASPs and SGPs
with loadsharing, broadcast and active/standby between ASPs and SGPs
as members of the cluster.  This is also what rserpool does.  Yet it
appears that you disagree with this approach.

What is required to support ASPs and SGPs in the model presented by
SIGTRAN is, indeed, the ability to fail over *between* associations.
Support for that is lacking in SCTP and the AS appears to side-step
the issue with it's statement that SCTP only supports failover within
a multi-homed association.

> 
> >>         In any event, a properly designed and implemented clustered
> >> system ought to look like a single system to the outside.  That is,
> >> there *ought not* be special protocol changes that would apply only
> >> to clustered systems.
> >
> >They why does SCTP not support this if special protocol changes *ought
> >not* be required?
> 
>         I did not say that the OS would not need modification.  The
> difference between old VMS and clustered VMS was that the OS got
> modified to support clustering.  DEC did not change on-the-wire TCP 
> when they added cluster support to VMS.  Similarly, on-the-wire SCTP
> ought not be changed when adding cluster support to some other OS.
> 
>         By definition, if SCTP explicitly supported this, then the 
> SCTP protocol would require "special protocol changes that apply only 
> to clustered systems".  

Does SCTP support failover between multi-homed hosts?

> 
>         It is up to whichever folks are designing the given clustered
> system and its OS to make the cluster look like a single system to
> the outside world.  Nothing in the SCTP specs prevents one from doing
> that in a myriad of ways (I can think of several approaches that could
> work fine).  The SCTP specs simply say that building a cluster is not
> within the scope of SCTP -- quite right, building a cluster is an OS
> and computer systems problem rather than a protocol problem.

Is failover between multi-homed hosts within the scope of SCTP?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---
You are currently subscribed to sigtran as: [email-masked]
To unsubscribe send a blank email to [email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 15:33:57 -0600

Allison,

So, is your answer no?

Is it fair to say that SCTP does not support failover between
multi-homed hosts, between associations, your second case?

Are you saying that applications such as SIGTRAN, RSERPOOL, AAA,
must implement their own sequence numbers and acknowledgements
and retransmission queues between associations?

--brian

Allison Mankin wrote:                                     (Sun, 04 Nov 2001 16:22:14)
> Brian,
> SCTP's multi-homing has no relation to anything between*
> hosts, only between interfaces of one host.  There might be
> some kind of cluster that appears to its SCTP peer be one host 
> with multiple interfaces, but it would not be within the scope
> of SCTP to know about the cluster. 
> 
> In draft-ietf-aaa-transport-04.txt, section 3.2. there's a 
> good discussion of failover/failback, which distinguishes
> among:
> 
> - what a cluster does in its cluster OS or cluster communication
>   system 
> - what application-level watchdogs do, where the failover is among
>   multiple potential peers (here AAA servers)
> - what SCTP does
> 
> If all interfaces of an SCTP host are down, that's the second case.
> 
> Allison
> 
> 
> ---
> You are currently subscribed to sigtran as: [email-masked]

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 15:53:14 -0600

RJ,

Is it the intention that  SCTP support what is being done by
those efforts without requiring them to institute application
-level acks, queues and retransmission for failover between
associations?

Should not SCTP's support (or lack of support) for SIGTRAN
protocols be mentioned in the AS?

As it stands it is not clear whether SCTP is even suitable to
transport the protocols that it was developed to transport...

--brian

RJ Atkinson wrote:                                             (Sun, 04 Nov 2001 16:32:40)
> At 15:58 04/11/01, Brian F. G. Bidulock wrote:
> >as members of the cluster.  This is also what rserpool does.  
> 
> I disagree with your assertion that defining how to build
> a cluster computer is what has been done by those efforts.
> 
> Ran
> [email-masked]
> 
> 
> 
> ---

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: RJ Atkinson <[email-masked]>
Date: Sun, 04 Nov 2001 17:48:49 -0500

>protocols be mentioned in the AS?

        I do not see that such would be strictly required.
SCTP is a general-purpose transport-layer protocol.

>transport the protocols that it was developed to transport...

        Disagree with your unstated assumption that SCTP exists 
only for SS7 purposes.  SCTP is designed as a general-purpose 
transport-layer protocol.  Its potential applicability is wide
and general.  For example, SCTP is being mentioned in some proposals 
in the multi6 WG as a potential way to support multi-homed hosts
more effectively in the IPv6 community.

Ran
[email-masked]

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 17:06:55 -0600

RJ,

RJ Atkinson wrote:                                             (Sun, 04 Nov 2001 17:48:49)
> more effectively in the IPv6 community.

I think you need a history lesson.

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: RJ Atkinson <[email-masked]>
Date: Sun, 04 Nov 2001 18:07:23 -0500

>I think you need a history lesson.

        Nope.  I know the history.  And I know that there was an
explicit decision made to design a general-purpose protocol (with
SCTP as the result).  I also know that some folks (perhaps you included)
wished that SCTP were ONLY for use with SS7.  That proposal for SCTP 
to be used ONLY for SS7 did NOT fly with the IETF community at large.  
In practice, I think SCTP will be used as much or more for applications 
other than SS7.  One of the Transport ADs has indicated that the
general-purpose nature of SCTP is part of the reason that SCTP 
was explicitly moved out of SIGTRAN and into TSVWG.

        Now one might not like that history, but dislike doesn't change
the facts on the ground...

Ran

---

---

From: Scott Bradner <[email-masked]>
Date: Sun, 4 Nov 2001 18:21:50 -0500 (EST)

> was explicitly moved out of SIGTRAN and into TSVWG.

almost but not quite

when SCTP was well under way, close to done in some people's minds,
Vern & I looked at it and thought it could be made into a more basic
and general purpose protocol by making some changes (over IP instead
of UDP for example) and we pushed to do that.  Doing the general purpose
protocol did not take away from the telephony signaling requirements

the decision to do this was not driven by the "IETF community
at large" - in fact we got some pushback from the wider community

(note that SCTP was never just for SS7 - it was for carying signaling
type traffic and used SS7 as one example)

finally - SCTP was moved out of sigtran after it was done because 
Allison and I felt that extensions to a general purpose protocol
should be done where we could be sure that they would be well
analyzed by the transport community - Vern and I got a reasonable
amount of tsv review by pushing tsv folk to look at it

Scott

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 17:33:42 -0600

RJ,

I would be happy if it *could* be used for SS7 as well, but it
has some problems, and its applicability appears to exclude SIGTRAN.

--brian

RJ Atkinson wrote:                                             (Sun, 04 Nov 2001 18:07:23)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: RJ Atkinson <[email-masked]>
Date: Sun, 04 Nov 2001 21:22:33 -0500

>has some problems, and its applicability appears to exclude SIGTRAN.

        I'm open to being persuaded that SCTP has specific problems, 
IFF someone would construct a specific detailed valid general scenario 
where SCTP is "broken".  So far, such a case has NOT been presented 
here.

Ran
[email-masked]

---

---

From: Allison Mankin <[email-masked]>
Date: Sun, 04 Nov 2001 21:41:38 -0500

Brian,

> So, is your answer no?

Yes, my answer is no :) - SCTP is not intended to fail over between
different associations - the protocol reliability is within
the association.  With the endpoints being on different boxes,
that is, in arbitrary network locations, it would be a different
protocol to do what you request.

But multiple SCTP associations can still be used as failovers
for each other, through use of an application layer watchdog and
a short list of duplicate transactions that might need to be sent.
The section on aaa transport that I recommended (3.2 in
draft-ietf-aaa-transport-04.txt) is about a robust way to do this.

> multi-homed hosts, between associations, your second case?
Yes, that's fair.  But to repeat, those different associations have
arbitrary endpoints.   How do SIGTRAN applications know what to make
of a set of message that start arriving suddenly because failover has
occurred - there must be application support of backup server use.

> and retransmission queues between associations?
No, I think SIGTRAN and RSERPOOL apps (and aaa, by way of the draft I
recommended you look at) can use multiple associations with an application
layer algorithm that is different (and simpler) than sequence numbers,
acks and transport retransmission queues.  

There needs to be duplicate detection of transactions, but failing over
within the SCTP association can result in duplicates too.  

I think you should look at the algorithm I've recommended and compare it
with the specific requirements.  

Allison

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 21:12:41 -0600

Allison,

A couble of remarks on the watchdog timer from AAA and its
(lack) of applicability to SS7 UAs:

1)  The timer is 30 to 60 seconds.  SS7 UAs must normally
    failover with 0.1-0.5 seconds.

2)  The watchdog timer is an application transaction timer.
    SS7 UAs have no implicit acknowledgement or transaction
    structure within the message flow that is usable by M2PA,
    M2UA, M3UA or SUA in determining when to cancel the
    watchdog timer.  Data in these UAs is unstructured (or
    at least its structure is not known to the UA).  A sequence
    number would need to be added to messages and acknolwedged
    separately to cancel the timer.

Because there is no structure know by the UA (no separate
transaction and response as there is in AAA), sequence numbers
and acknowledgements would be required for each message.
Then the watchdog timer reduces to a retransmission timer.

So, it appears that you are saying that SIGTRAN protocols must
implement sequence numbers, acknowledgements, queueing and
retransmission after all.

Are you sure this is what you wish to recommend rather than
fixing several problems in SCTP?

--brian

Allison Mankin wrote:                                     (Sun, 04 Nov 2001 21:41:38)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Sun, 4 Nov 2001 21:21:41 -0600

RJ,

It has been presented here (TSVWG and SIGTRAN) and will be presented
again:

For failover between associations the ULP needs to be able to
do the following things:

1)  Withhold acknolwedgements at ULP control for intervals longer
    than several seconds without aborting or otherwise degrading
    an SCTP association.  RFC 2960 says that all new DATA messages
    MUST be acknowledged and sets upper bounds (usually 200ms) on
    any acknowledgement.

2)  Cause the remote peer to stop transmitting new data.  RFC 2960
    permits an association to send new DATA regardless of whether
    it has receive a_rwnd = 0.

3)  Have sent messages acknowledged or have the ability to retrieve
    unsent and unacknowledged messages after an ABORT.  This is
    currently possible and is largely the responsibility of the
    implementation.

Failure to provide (1) and (2) means that an ULP cannot stop the
movement or clearing from the peer's retransmit buffer messages
that cannot be processed locally.

So, please fix those two problems with SCTP so that it can be
used for SIGTRAN UAs.  Solutions have even been proposed: a solution
to (1) is the continguous Gap Ack approach that was dicussed on this
list several months ago.  The solution to (2) is to enforce that
the sender MUST NOT send new data when rwnd = 0 and can only use old
data (or an acked TSN) to "probe" for lost SACKs.

Have any problems with that?

--brian

RJ Atkinson wrote:                                             (Sun, 04 Nov 2001 21:22:33)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 07:03:58 -0600

Brain:

Comments below..

> a multi-homed association.

Ahh.. so the ASP's still do exist in the architecture.. then this
can solve your problem with small help from the transport layer 
giving you what was sent and queued... when a failure occurs..

> > ought not be changed when adding cluster support to some other OS.

It should not have to be. see below..

> Does SCTP support failover between multi-homed hosts?

I think there is fundamentally a problem in this dicussion on
what a "cluster" is.

A)Is a cluster a group of computers that give 
an appearance to the outside internet has a
single computer.

or

B)Is a cluster a logical group of processes scattered around
the internet providing a service.

In the latter case I think this is what the ASP concept is
about and it is ALSO what RSERPOOL is all about.

In the former case you can roll whatever you want to do clustering.
I don't see you needing to standardize how you build up multiple
computers that looks like a single machine to the outside world.
You can do what you like i.e. you can take the Tandem triplex
voting mechanism or anything else you dream up. If it truely
looks like a single computer then SCTP and TCP should work
just fine across all the components as well.

> Is failover between multi-homed hosts within the scope of SCTP?

If you are defining the cluster as in (A) above, then SCTP will
work just fine.

If you are defining the cluster as in (B) above, then this is
what the ASP and RSERPOOL concept is for. You break the problem
in to pieces. SCTP cannot be easily written to keep transport
state synchronized across the Internet between two machines. Instead
you put in your PE/PU concept (or ASP concept for SIGTRAN) that
divides the problem to make it more managable..

R

> --brian
> 
> --
> Brian F. G. Bidulock
> [email-masked]
> http://www.openss7.org/
> 
> _______________________________________________
> tsvwg mailing list
> [email-masked]
> http://www1.ietf.org/mailman/listinfo/tsvwg

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 07:21:12 -0600

>     MUST be acknowledged and sets upper bounds (usually 200ms) on

Yes, and has Tom George pointed out one can do as mentioned by
NOT advancing the cum-ack and putting new chunks in GAP acks.
This of course penalizes our cwnd growth since the cwnd is
not allowed to grow unless the cum-ack advances but this
should not be a problem for sigtran. And there may be some
implementations that abort when this occurs.. but it is not
wrong so you should NOT abort really..

>     it has receive a_rwnd = 0.

As I mentioned before this is a arguable point but I will gladly
add the clearification to the implementors guide ..

>     currently possible and is largely the responsibility of the

The socket api that is a wg item allows this.. I do need to
get the latest version in though... 

So I don't see any of your problems as problems.. they all
have been addressed..

R

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 08:07:22 -0600

Randall,

Randall Stewart wrote:                                    (Mon, 05 Nov 2001 07:21:12)
> So I don't see any of your problems as problems.. they all

So we will see both (1) and (2) clarified as required in the
"implementor's guide" or whatever follow-on RFC?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 08:18:23 -0600

> "implementor's guide" or whatever follow-on RFC?

I don't know about (2) since it is already allowed behavior
but for sure we can clearify (1) above..

R

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 08:59:14 -0600

Randall,

Randall Stewart wrote:                                    (Mon, 05 Nov 2001 08:18:23)
> but for sure we can clearify (1) above..

(2) is clearly not allowed behaviour due to the MUST in
Section 6.2 that says:

    "In paticular, the SCTP endpoint MUST fill in the Cumulative
     TSN Ack field to indicate the latest sequential TSN (of a
     valid DATA chunk) it has received."

In fact, in the mail threads cited by David, the concensus was
that this was a "protocol violation".

What makes it "allowed behavior" now?

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 09:22:21 -0600

> What makes it "allowed behavior" now?

Ok, its not allowed behavior.. I still don't understand the problem.

A lot of implementations it will work.. I don't think most
will go to the trouble of checking this.. I know I did not
in any implementation.

But my base problem is why is this needed? If you are thinking
in terms of per-stream-flow control I believe the next pass
of the add-ip draft is supposed be updated to either

a) drop the sender flow limits 

and

b) (hopefully if Phil will provide the text :> this is again
    a nudge Phil) provide a per-stream flow control that 
   satisfy's this behavior.

--
R

---

---

From: Phillip Conrad <[email-masked]>
Date: Mon, 05 Nov 2001 10:56:38 -0500

A bit of comparison between TCP and SCTP may be in order here.  In fact, I 
would suggest that at least one of Brian's statements about TCP behavior is 
in direct conflict with RFC793, as I explain below.   However, on a least 
one point, I would argue that Brian and Randall have come close to reaching 
consensus, although amid the smoke and flame it may be hard to perceive.

Randall Stewart proposed the following change:

>Note: Anytime the receivers rwnd is 0, the receiver MAY drop
>the DATA chunk without acknowledgment (since it has no room
>to receive the data nor validate it).

"Brian F. G. Bidulock" further noted:

>M2PA (and others) would like to use
>artificial lowering of a_rwnd to 0 as a way to stop the sender
>from sending new data or at least stopping transfer of new data.
>Yes, there is lots of real buffer space.  a_rwnd is for flow
>control and setting it to zero never means that all physical
>memory has been exahausted.  (You will see an ABORT very quickly
>after that happens).
>
>It appears that everything is devised to not require that
>data flow stop (as in TCP).  Who cares if some broken implementation
>that leaves a_rwnd = 0 and acks everything gets stuck?

However, RFC793 notes the following (section 3.7, p. 41)

>The sending TCP must be prepared to accept from the user and send at least
>one octet of new data even if the send window is zero.

Thus, in both TCP and SCTP it is possible to send new data on a zero 
advertised window.   So I think Brian's statement is in conflict with 
RFC793: the data flow does not "stop", if "stop" means that no new data can 
go on the wire.   However, in a larger sense, Brian raises a good point, 
because RFC793 continues:

>   ... The
>   sending TCP must regularly retransmit to the receiving TCP even when
>   the window is zero.  Two minutes is recommended for the retransmission
>   interval when the window is zero.  This retransmission is essential to
>   guarantee that when either TCP has a zero window the re-opening of the
>   window will be reliably reported to the other.
>
>   When the receiving TCP has a zero window and a segment arrives it must
>   still send an acknowledgment showing its next expected sequence number
>   and current window (zero).

So far, TCP and SCTP are in agreement at least to this extent... when presented
with a zero window, it is necessary for the sender to "probe" the window with
segments that may consist of new data.   A difference is that in TCP, these
window probes are a single byte, while in SCTP, they may consists of an entire
data chunk.

The key difference however, is that in TCP, as I understand it, the 
receiver will NOT accept
or acknowledge the NEW data (i.e. the window probe) if the advertised window
is still zero when the new data reaches the receiver.  While I could not 
find text in the
RFC that spells this out (can someone help here?) I did find a nice 
explanation and
example in W. Richard Stevens, TCP/IP Illustrated Vol 1, section 22.2, p. 
323-324.

If SCTP doesn't ALSO stubbornly refuse to acknowledge the new data that is sent
when the a_rwnd is zero, well, then I would agree that SCTP is broken in 
that regard.
If new data is sent when the a_rwnd is zero, then this new data should be 
considered
a "window probe".  The correct response to a window probe, if the a_rwnd is 
still zero,
should be to ignore the new data, and send a repeat of the previous 
acknowledgement
state ... the cum ack value, in the case of TCP, or a repeat of the 
previous SACK chunk
in the case of SCTP.

Now, it seems to me that this is exactly what Randall is proposing, though 
I would suggest
that the MAY should become a MUST, as shown in the square brackets:

>Note: Anytime the receivers rwnd is 0, the receiver MAY [MUST!] drop
>the DATA chunk without acknowledgment (since it has no room
>to receive the data nor validate it).

If the receiver DOES accept the DATA chunk
and ack it, where the heck is it supposed to go?    Unless the receiver has 
a "secret stash"
of memory beyond the advertised window, there isn't any place.  And, to NOT 
stop receiving
(so that, if the sender implements a blocking API, the backpressure can 
reach the sender)
is misleading to the sending application, and should be considered harmful.

Can we reach consensus on at least this ONE small point?

Phill Conrad

Phill Conrad -- Asst. Professor, CIS Dept, Temple University

NOTE: For faster replies, please reply to [email-masked], NOT 
to [email-masked].

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 09:53:57 -0600

Phillip,

It is the "MUST" behavior that we are looking for.  However, what
happens to DATA chunks in flight that have RTO timers running
against them.  It takes time for the a_rwnd = 0 SACK to reach the
peer.  Also SCTP has path and association retransmission counts
and congestion window calculations that should not be performed
on the "probe" data chunks.  There are a whole series of MUSTs
concerning these counters and calculations.  These should be
suspended upon calculating a rwnd = 0 and the probing behavior
started.  Once rwnd permits new data, cwnd and couters can
continue.  Otherwise, a_rwnd = 0 will degrade the association
(lower cwnd, back off RTOs).  What is required is simple flow
control.

--brian

Phillip Conrad wrote:                                     (Mon, 05 Nov 2001 10:56:38)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: Phillip Conrad <[email-masked]>
Date: Mon, 05 Nov 2001 12:04:30 -0500

>(lower cwnd, back off RTOs).  What is required is simple flow

I agree that simple flow control is needed, and yet, without a window-probe 
function, the protocol may deadlock.

It sounds like what is needed is a clear concept of a "window probe" that 
is treated differently in a wide variety of respects from regular data 
chunks.  What may be needed is to carefully spell out the rules pertaining 
to "window probes", and handle the special cases to which you refer so that 
window probing does not adversely affect cwnd, RTO, etc.    A window probe 
should not be treated according to the regular retransmission timeout 
rules, but should be handled with a persist timer, as in TCP.

RFC2960 has the concept of refraining from sending on a zero window, and 
the concept of a window probe:

>A) At any given time, the data sender MUST NOT transmit new data to any 
>destination transport address if its peer's rwnd indicates that the peer 
>has no buffer space (i.e. rwnd is 0, see Section 6.2.1). However, 
>regardless of the value of rwnd (including if it is 0), the data sender 
>can always have one DATA chunk in flight to the receiver if allowed by 
>cwnd (see rule B below). This rule allows the sender to probe for a change 
>in rwnd that the sender missed due to the SACK having been lost in transit 
>from the data receiver to the data sender.

However the concept does not seem to appear anywhere else as a special 
case, which I think, is part of what you are trying to get across to 
us.  For example:

>Whenever a transmission or retransmission is made to any address, if
>    the T3-rtx timer of that address is not currently running, the sender
>    MUST start that timer.  If the timer for that address is already
>    running, the sender MUST restart the timer if the earliest (i.e.,
>    lowest TSN) outstanding DATA chunk sent to that address is being
>    retransmitted.  Otherwise, the data sender MUST NOT restart the
>    timer.

It should say something like: "Whenever a transmission or retransmission 
{other than a window probe} is made to any address..." Then, there should 
be explicit reference to a persist timer as in TCP that is used for window 
probes instead of the normal retransmission timer.

Oddly, I was unable to find a reference to a persist timer in any of the 
RFCs (either TCP or SCTP)... can anyone help here?  I looked through RFC793 
as well as Clark's later treatise on the Silly Window Syndrome (RFC813), 
but no luck.

Stevens' text indicates that the persist timer is different from the RTX 
timer "in that TCP *never* gives up sending window probes." (p. 325).

Adding a separate persist timer and rules for handling window probes is a 
fairly significant change to the protocol, but one that is necessary I 
think.  I had assumed all along that a zero-window would be handled in SCTP 
in precisely the same was as in TCP, but it appears not to be the case from 
the way things are worded now. Two of the issues:

* Is there some reading of RFC2960 that would indicate that we DON'T have 
to take a timeout (and all the associated bad congestion control effects 
that follow) in the event that we hit a zero window with new data?

* I'd also think we'd want to avoid, in a multi-homed situation, failing 
over to an alternate address with retransmissions if the "problem" is 
clearly that the receiver has a zero window.    Is there text in the RFC to 
handle this case?    Repeated window probes should probably stay on the 
primary as long as acks are coming back from the primary... acks that in 
this case, DON'T ack the new data, but rather, just indicate that the 
advertised window is STILL zero.

(I don't think the introduction of per-stream flow control can make *this* 
problem go away; window probes are still needed to ensure forward progress)

Phill

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 11:41:44 -0600

Replying once to both :>

> >(lower cwnd, back off RTOs).  What is required is simple flow

I am not sure I see the value of seperate MUST behaviours for
all the other things. If I am allowed to drop the chunk
but still respond with a SACK (showing no movement in anthing), I
don't see how anything will change except that if the sender is
blocked long enough you may get a T3-rxt. If this occurs you
do collapse your cwnd. This I think is OK considering that 
you have not really been sending much so the ACK clock is lost
anyway so one must slowly probe the network for bandwidth and
slow start IS appropriate!

> rules, but should be handled with a persist timer, as in TCP.

And why not? If the peer drops all inflight (while the rwnd of 0 is
being sent back) and then sacks the dropped chunk everyother one..
Assuming that the app is NOT reading and the rwnd does not lower
what will surely happen is that eventually nothing is outstanding
rwnd is set to 0 and it will not be until the t3-rto that we
send out something. At that point you have lost your ACK-Clock and
all knowledge of what the condtions of the network are. I would contend
that if the reader is being stubborn then the association (once it
finally does read) will need to go through a slow-start process
back up to the sending rate it was at when the receiver halted reading.

I don't see this has a bad thing and it IS along the lines of what
TCP does I believe...

> as well as Clark's later treatise on the Silly Window Syndrome (RFC813),

I don't know if this exists.. I have never found it either.. I think
it is left unspecified...

> advertised window is STILL zero.

The multi-homing issues may be one we want to explore, but I don't
know if the cwnd collapse is a bad thing.

Does TCP stop the t3-rxt in a window-probe situation? This is
the bottom line to what you are getting at... I took a
quick look at Stevens as well.. and all of its illustrations
are when you have ALL data outstanding acknowledged when the
0 lenght window arrives... What we are talking about here
is when you do

-----send(1-mtu)--------------------------
                           /-------send sack rwnd=0
-----send(1-mtu)---\      /
                    \    /
                     \  /
                      \/
                      /\
                     /  \
<--------------------    ----------------------->

And so we still have a bunch of data outstanding. This
means the t3 is still running.. I would bet in most TCP
implementations it would still be running as well thus
you WOULD get a cwnd collapse in this case.

If you are just probing i.e. you are sending with
a rwnd=0 and nothing is outstanding.. then in some
respects I can see running the seperate probe timer etc..
makes sense .. but you HAVE lost the overall ack clock. Even
in Stevens it talks of the probe timer growing exponentially..
this means you don't send another probe for a longer and longer
time and thus slow-starting is in my mind a good idea.

> problem go away; window probes are still needed to ensure forward progress)

I agree here Phil.. and I am still waiting for some text :>>

R

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 12:24:56 -0600

Phillip,

I agree largely with your comments.  Particularly that separate
probe procedures are called for.

However, I have seen TCP implementations (BSD, Linux) that do
not send new data on probes, but rather send a duplicate segment
with bogus data (or zero length segment) on probes.

I don't see any reason why SCTP cannot send a DATA chunk with
a duplicate TSN (with bogus data) to invoke an immediate SACK
(thus getting rid of the lost SACK problem).  The remainder is
giving the sender the ability to ignore RTOs, counts, cwnd calcs,
etc during the probe.  An upper bound on the probe duration would
be smart (persist timer?).

But how does one go about getting all this into SCTP?

--brian

Phillip Conrad wrote:                                     (Mon, 05 Nov 2001 12:04:30)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 12:43:18 -0600

Randall,

Randall Stewart wrote:                                    (Mon, 05 Nov 2001 11:41:44)
> slow start IS appropriate!

I disagree that the The T3-rxt timeout is appropriate.  The network
has not lost a packet.  The probe was received fine.  SCTP does not
close cwnd and go into slow start when traffic slows, why should it
when traffic stops (except for probes)?

> I don't see this has a bad thing and it IS along the lines of what

It is a bad thing as it has degraded the association when the only
thing that has happened is flow control (not congestion).  SCTP should
not misread flow control for congestion.  We could keep the ack clock
going by sending DUP chunks and cancelling RTO off the DUP SACKs with
the appropriate DUP probe chunk.

> know if the cwnd collapse is a bad thing.

It is a bad thing for M2PA.  Artificially dropping a_rwnd to zero
can be a transient thing and the association should not enter slow
start or back off where there are no indications of network congestion
(just user flow control).

> -----send(1-mtu)--------------------------

If you consider the view from the application, it is a disasterous
idea.  The requirement is for the application to be able to stop
the data flow (at least for short intervals, several seconds) without
degrading the association.

--brian

> You are currently subscribed to sigtran as: [email-masked]

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Ong, Lyndon" <[email-masked]>
Date: Mon, 5 Nov 2001 11:00:55 -0800

If this probe is a single-end process, it should be
possible to add this as some kind of extension,
would it not?  BTW, shouldn't this discussion be
on the tsv list rather than sigtran?

Thanks,

Lyndon

-----Original Message-----
From: Brian F. G. Bidulock [mailto:[email-masked]]
Sent: Monday, November 05, 2001 10:25 AM
To: sigtran
Cc: Randall Stewart; [email-masked]; RJ Atkinson;
Phill Conrad
Subject: Re: [Tsvwg] SCTP applicability statement

Phillip,

--brian

Phillip Conrad wrote:                                     (Mon, 05 Nov 2001
12:04:30)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---
You are currently subscribed to sigtran as: [email-masked]
To unsubscribe send a blank email to
[email-masked]

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 13:09:30 -0600

Lyndon,

It is M2PA interop results that started this off.  M2PA needs
a solution here or it will have to introduce sequence numbers
and acks at the ULP level to meet MTP3 requirements for
conformance.

--brian

Lyndon Ong wrote:                                          (Mon, 05 Nov 2001 11:00:55)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 13:24:22 -0600

Brian:

Take a close look at RFC2960... you will find that
you are required around every RTO or so to degrade
the cwnd when you are not actively sending data. In
the case you describe where we are doing window probes
you end up backing off ... take a look at Stevens
TCP Illustrated Vol 1 chapter 22... the probe
backs off as well..

This means you send less and less often. You lose
the ack-clock in the process.. yes it is due to
flow control not congestion control.... but this
does not matter once you lose the ack clock you
must be cautious about restarting the flow of
data. Slow start is in order and a SLOW probe
for available bandwidth I think is a must..

R

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 13:28:12 -0600

Brian:

Phil has promised me a stream level flow control real soon
to augment the add-ip draft and replace the stream flow
limits stuff... why don't you hold off to give Phil
a chance to describe the flow control in the upcoming
add-ip document...

Phil, all eyes are on you now.. no pressure :>

R

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 14:36:12 -0600

Please see page 88 last bullet point:
" 
   o  When the endpoint does not transmit data on a given transport
      address, the cwnd of the transport address should be adjusted to
      max(cwnd/2, 2*MTU) per RTO.
"

"Brian F. G. Bidulock" wrote:
> 
> Randall,
> 
> Randall Stewart wrote:                                    (Mon, 05 Nov 2001 13:24:22)
> > Brian:
> >
> > Take a close look at RFC2960... you will find
> > that you are required around every RTO or so
> > to degrade the cwnd when you are not actively
> > sending data.
> 
> No, I don't see that.  Please provide a reference.
> It appears to me that idle destinations are sent
> heartbeats and that if those heartbeats are ack'ed
> no cwnd ajustment will result.  Slow start must
> be peformed after a "sufficiently long idle period".
> 
> > In the case you describe where we are doing
> > window probes you end up backing off ... take
> > a look at Stevens TCP Illustrated Vol 1
> > chapter 22... the probe backs off as well..
> 
> Did I give you the impression that I thought TCP was
> suitable for M2PA?  It is not.
> 
> >
> > This means you send less and less often. You
> > lose the ack-clock in the process.. yes it is
> > due to flow control not congestion control....
> > but this does not matter once you lose the ack
> > clock you must be cautious about restarting
> > the flow of data. Slow start is in order and a
> > SLOW probe for available bandwidth I think is
> > a must..
> 
> Certainly as stated in the RFC this is not
> required until a "sufficiently long idle period"
> has elapsed.  With no data on the connection, HB
> should keep RTT trimmed and cwnd stable,
> regardless of whether the ack clock stops.
> There is no need to run T3-rtx, just as there is
> no need to run T3-rtx if there is no data
> outstanding acknowledgement.
> 
> --brian
> 
> >
> > R
> >

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 14:49:37 -0600

Randall,

I though you meant "required".  There are no RFC keywords in
this bullet point and the statement even says "should".  Its
pretty clear that an implementation can pretty much ignore that
one.  It it wants to mangle the association right away, and
not wait for a "sufficiently long idle period", that's its
problem.

Nevertheless, there is no requirement to back off during
idle periods or probes, ack clock or not.

Is there a requirement in TCP?

--brian

Randall Stewart wrote:                                    (Mon, 05 Nov 2001 14:36:12)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 14:23:40 -0600

Randall,

--brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 05 Nov 2001 17:10:38 -0600

...snip...

Brian,

It seems to me that if you use a ULP level acknowledgment, this problem
goes away, right? If so, all you are trying to do is to push more app
specific functions down into SCTP so that the ULP can get simplified. I
think that is wrong.

-Qiaobing

---

---

From: Randall Stewart <[email-masked]>
Date: Mon, 05 Nov 2001 17:12:19 -0600

Brian:

Thank you for pointing out the weak wording here. We can 
add this to the list of things to fix in the implementors
guide... 

As far as TCP is concerned, I believe some or Mark Allmans
proposed extensions (I think ABC) have this in place.

R

---

---

From: Qiaobing Xie <[email-masked]>
Date: Mon, 05 Nov 2001 17:35:05 -0600

Brian,

This is the kind of common misconception flying around often, that if
you use a reliable transport (TCP or SCTP), then you no longer need to
worry about app-level timers, acks, retrans, heartbeats, integrity
checks, etc., since all those will be done by the transport.
Unfortunately, this is false.

But this is the first time I've heard anyone saying you can even get rid
of app-level queues by using a reliable transport.

> protocols be mentioned in the AS?

SCTP never intends to support doing EVERYTHING for SIGTRAN. Instead, it
just provides the "SIG" layer for SIGTRAN as stated in RFC2719. ULPs
will need to do their part as well.

-Qiaobing

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 17:51:08 -0600

Qiaobing,

Quite frankly I agree with you.  I would prefer an application level
sequence number, ack, queuing and retransmission.  Or, basically what
you term buffer management.  I have proposed on several occasions that
M2PA and other use a sequence number tagging and acknowledgement to
conquer this difficulty.  M2UA does have a Correlation Id/Ack which can
be used to the purpose of flow control an buffer management between M2UA
peers.

We started discussing this back in May under the subject "M2PA:
Acknowledgement Problem".  (Probably even further back.) I proposed that
an sequence number and ack be added to M2PA to achieve this flow
control.  We also went through a lot of discussion on TSV regarding the
retrieval of messages using the SCTP sockets draft.

The advantages of an application level flow control is that it permits
application flow control to be completely independent of SCTP
parameters.  It will be difficult enough tweaking SCTP parameters,
buffer sizes, RTO.min, max, retrans maxes, and other characteristics of
SCTP to best suit the network over which it is running without messing
it all up with application flow control.

If TSV can say that "SCTP can't (or won't) do this" then there are other
solutions at the application layer with a wider range of benefit.

The level of resistance in SIGTRAN to placing application level flow
control in the xA has boistrous:

	"till now we don't have sequence number at m2pa
	level.  and it doesn't make any sense to make queue
	at m2pa level."

	"Talking of queues at M2PA level suggests that we
	are trying to mimick MTP2, which in my opinion is
	incorrect as we already have reliable service from
	SCTP as transport protocol for M2PA."

	"I see no value addition to the M2PA Stack by adding
	Seq.  Numbers, moreover i see an additional
	overhead."

	"Since M2PA has an Already proven and reliable
	medium.  I do not find any positive traits by adding
	SSn to the M2PA Stack."

	"I also think that ACK functionality at the m2pa
	level is redundant overhead."

	"Why can SCTP not withold its ACKs?"

	"Is there some reason why the SCTP spec could not be
	modified so that it allowed the ULP to withhold
	ACKs?"

	"I think the problem should be addressed at the SCTP
	level.  Today we have a requirement in M2PA,
	tommorrow the same may come up in some other layer.
	Also SCTP-SCTP reliability is useless if
	SCTPULP-SCTPULP reliability is not guaranteed."

	"Is there a previous thread where m2pa level
	buffering of user messages was discussed and a
	conclusion made that support at SCTP level is
	insufficient or not necessary?"

	"..."

And on, and on: I just went back on one short thread.

Either SCTP supports this capability or it doesn't.  Please pick and for
mercy sake write it in the AS.  I prefer that TSV say "No: SCTP does not
support this (it has enough on its plate), it is the application's
responsibility to institute application flow controls to the degree
necessary to provide redundancy across associations."

--brian

Qiaobing Xie wrote:                                      (Mon, 05 Nov 2001 17:10:38)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 17:55:14 -0600

Qiaobing,

As I said (although more long-winded) in my previous note, please
set SIGTRAN straight on this matter and write in no uncertain terms
that these are the responsibility of the ULP.  Then we just add
acks, retrans and ULP flow control to M2PA and it will run like a top.

You are actually preaching to choir.

--brian

Qiaobing Xie wrote:                                      (Mon, 05 Nov 2001 17:35:05)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Mon, 5 Nov 2001 14:26:24 -0600

Randall,

I suppose it is up to the WG to decide whether they
want M2PA blocked until someone descides to complete
an optional extension draft which would be difficult
for M2PA to mandate.  I agree with Phillip that It
would be better if SCTP could be fixed.

--brian

Randall Stewart wrote:                                    (Mon, 05 Nov 2001 13:28:12)

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

---

From: David Lehmann <[email-masked]>
Date: Tue, 06 Nov 2001 08:44:04 -0500

> necessary to provide redundancy across associations."

Amen!

> > specific functions down into SCTP so that the ULP can get simplified. I

Amen again!

-David

---