Subject: Re: [Tsvwg] sctpsocket: Returning unsent messages

From: Kacheong Poon <[email-masked]>
Date: Tue, 26 Jun 2001 15:36:23 -0700 (PDT)

> > > Maybe Brian or other people familiar with the sigtran protocols
> > > can comment on this.  Do we really need to separate the sack'ed
> > > but not cumulatively ack'ed case?  The reason I ask is that the
> > > receiver is free to discard sack'ed but not cumulatively ack'ed
> > > data chunks.  And I believe it is better not talk about TSN in
> > > the API draft.  It seems to me that making this distinction is
> > > not that useful.  I maybe wrong.  I suggest we should include them
> > > both as SCTP_DATA_SENT?  This has a easier to understand semantics
> > > for app programmers without the need to understand SCTP fully.
> > >
> > 
> > I admit the difference is slim. The only reason I suggested it
> > was the maybe apps we haven't thought of yet might actually
> > find it useful.
> > 
> > Jonathan, easily swayed on this one.
> > 
> 
> I see no reason NOT to add the SACKED as well. The information is
> there, you are already passing up chunk status.. why not
> have it passed up as well.

The main reason I think is that it is a good idea not to expose protocol
details to apps if possible.  And as I pointed out previously, something
sack'ed really does not mean much.  Data can be discarded or data
may not be able to be delivered to the app because of ordering issue.
So this is kind of ambiguous.  Brian and other have pointed out that
there is an app level control to deal with this ambiguity if an app 
chooses to do this retrieval.  I don't see why we need to complicate
the API for the little ambiguous info.

							K. Poon.
							[email-masked]

_______________________________________________
tsvwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 26 Jun 2001 21:10:30 -0500

Kacheong,

Kacheong Poon wrote:                          (Tue, 26 Jun 2001 15:36:23)
> the API for the little ambiguous info.

Well its a little late to start stopping the exposing protocol details.

It may, unfortunately, be necessary for some application to know which
messages may have been delivered to the user on the other side.  With
multiple streams the only way to determine this at present is to check
which sacked messages contain contiguous SSNs for the streams in question.
Also, an application may want to do retrieval for some streams but not
for others.  I don't know how to provide this information across a
socket API without exposing some SCTP protocol details.

--Brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 27 Jun 2001 21:37:08 -0700 (PDT)

> socket API without exposing some SCTP protocol details.

Such apps need to have their own protocols.  This is impossible for
SCTP to tell them which messages have been delivered.  Even a message
is sack'ed and in the correct order of its stream, it does not mean that
it is delivered.  All data reported by SCTP as sent can be potentially 
delivered to the other side.  The apps have to be prepared for this and
it will be written down in the draft, although personally I won't
recommend using it.  I still don't see the reason why telling the apps the
incorrect and complicated info can really help them.  If the apps make use 
of this API feature, it has to have its own protocol, regardless of whether
the API exposes the SCTP sack info...

							K. Poon.
							[email-masked]

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 28 Jun 2001 01:11:46 -0500

Kacheong,

I agree with you 100%.  Such apps should have their own protocols.
I would like to see all apps (and UAs) have their own app-to-app
acknowledgements and handling of aborted associations to avoid
message loss or mis-sequencing across multiple associations.

However, current proposals for acknowledgement elements in M3UA have
not been adopted and have been poorly received.  M3UA may likely
have a Broadcast mode added which provides the protocol necessary to
avoid these complications of retrieval.  However, for the other two
modes: Override and Loadshare, there are still problems.  Without
the simple mechanisms allowed by an application level ack (which has
currently been rejected as "over-kill"), M3UA in Override or
Loadshare mode might have to determine which messages were
"possibly" delivered to the peer M3UA so that it can discard them to
avoid duplication (at the cost of message loss).  In determining
whether messages where "possibly" delivered, information concerning
whether the messages were gap-SACKed or not could be useful to the
implementation in determining which messages are best to discard
and which are best to transmit via another route or ASP.

I agree with you that the cleanest way to support this is to have
an application level ack and I have tried to persuade M3UA to
include this (my pesuasion skills are not very good ;).

All of the UAs are in the same boat.  Some are considering my
proposals for an application level ack to avoid the ambiguities
inherent in SCTP retrieval.

The best approach, I believe would be to remove retrieval from the
API altogether.  It is not needed by "normal" applications (like
HTTP) which provide their own acknowledgements at the application
level.  This might disuade the SIGTRAN UAs from attempting to abuse
the interface and require them to produce protocols which meet
their specific needs rather than trying to paint them all with the
same brush with one mechanism in SCTP.

--Brian

-- 
Brian F. G. Bidulock
[email-masked]
http://www.openss7.org/

---

From: Randall Stewart at work <[email-masked]>
Date: Thu, 28 Jun 2001 09:03:02 -0500

Kacheong/Brian:

I do not agree :) 

I am not sure I need the details of if it was 

SACK'd vs transmitted vs not-ever-transmitted.

But I can envision an application using if it was

transmitted versus not-ever-sent.

Imagine a application DOES implement a protocol to
find out, with its peer, if certian messages have
been sent. This sequence is long and costly and 
requires multiple messages, database searches etc on
the remote peer. And we may be on some aux link where
you want to minimise traffic too..

Now if I have 100 messages in queue, and only 5 were
ever assigned a TSN and sent... If I know that I can
save a tremendous amount of overhead on those 95 messages
that were never sent... I KNOW they were not sent this is
very useful information.. The other 5 .. no matter if they
were SACK'ed or not will need to have the audit procedures
run on them...

I think retrieval is a very good thing and does not cost
much to implement. I do want it to stay in the API. 

R

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 28 Jun 2001 12:11:30 -0500

Randall,

The chance of abuse by SIGTRAN UAs is very high.  Consider
that SSCOP, NSD and SS7 L2 all provide application level
acks.  Unlike SCTP, these protocols do not require ACKs
for RTT calculations, so the sent messages are only acked
once they have been delivered to the ULP.

These protocols use the sequence number for communication
between each end for coordinated retrieval and retransmission
on an altenate path.  They do not require database lookups
or any heavy mechanisms.  The mechanism is built into the
acknowledgement procedure.  (Again, the ack is only sent
once the message has been delivered to the peer app.)

When the communication fails abruptly, the sender has in
its retransmission buffers messages which may have been
already delivered to the peer app, but for which the
ack has been lost.  It is a simple matter that if the
peer apps can communicate by other means, that they indicate
to each other the sequence number of the last message
successsfuly delivered to the app.  The sender strikes
these messages from its retransmit buffer before retrieval
and resends these messages on an alternative path.

SCTP cannot provide this mechanism on its own unless we
get into the gap-acking mechanism we discussed a while ago.
SCTP acknowledges and clears from the sender's retransmission
buffer messages which may not have been delivered to ULP.
For the retrieval mechanism to work, the sender cannot strike
the messages until they have been successfully received
by the peer application (rather than just SCTP).

It is fairly straightforward for the sending app to number
all messages which are subject to retrieval and have the
receiving app acknowledge messages actually received at
the app by message number.  This effectively moves the
retrieval process above the SCTP-ULP interface line.

Attempting to use the SCTP retrieval process at the application
level will result in the possibility that messages need be
marked for retransmission on the new application path which
have already been struck by SCTP.  To avoid this message
loss the application will have to buffer messages in a 
retransmission buffer anyway and performing the ack mechanism
at the application is trivial.

Some other applications of SCTP might want retrieval; however,
the SS7 SIGTRAN UAs will not be able to use it.  Can anyone
think of an application other than SIGTRAN UAs which will
want retrieval and what the semantics of that retrieval will
be?

--Brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Randall Stewart at work <[email-masked]>
Date: Thu, 28 Jun 2001 13:48:31 -0500

Brian:

When I said DB applications I was hinting at applications
beyond sigtran. Let me illustrate an application or
more correctly a session layer that uses this feature.

When our RSERPOOL/ASAP implementation sends it
populates the context field with the send flags.

If the peer dies, the data is bounced up to the
ASAP layer since it enables this retrieval flags.

The context is consulted to see if the send allows
re-routing the packets to a alternate peer.

If so, the ASAP layer re-selects an alternate peer
and retransmits the data to it.

The retrieval mechanism is quite useful in this case and
is needed by the session layer namely ASAP.

The whole reason I added the context field in was for
this very reason :) You will note it IS in RFC2960.

Now I realize you are coming from a SS7 layer world, but there
ARE other applications for this data... and knowing the
difference as to if it HAS been sent OR HAS NEVER been
sent is also a nice added bonus of the API proposal..

R

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 28 Jun 2001 12:46:33 -0700 (PDT)

> SACK'd vs transmitted vs not-ever-transmitted.

Hmm, Randy, I think you misread the flow of the discussion.  We 
were discussing whether we should distinguish sent and sack'ed from
sent but not sack'ed data...  And then we were discussing what the
meaning of retrieving data for a normal shutdown...  We were not
discussing whether we need to put in the mechanism to retrieve data
for a send failure case...  The mechanism is already there.

							K. Poon.
							[email-masked]

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 28 Jun 2001 13:36:41 -0700 (PDT)

> is needed by the session layer namely ASAP.

Randy, can you say more about this?  So does the context
indicate whether the message can be duplicated?   And I
believe the retrieval is done after SCTP gives up.  The 
reason I ask is to find out if ASAP needs only unsent
messages.  I think this is the whole point of the current
discussion.  What is useful and what is not useful?  And
do we need to expose SCTP's SACK info?  To me, I think
only unsent messages is useful.  If the app has a duplicate
protection mechanism, then sent messages may be useful.
And I don't see the need to distinguish sack'ed and not
sack'ed messages.  Then Jacob proposed to include an user
induced failure behavior.

							K. Poon.
							[email-masked]

---

From: Randall Stewart at work <[email-masked]>
Date: Thu, 28 Jun 2001 17:44:16 -0500

> for a send failure case...  The mechanism is already there.

My impression from Brian's comments is that he would
like the mechanism removed... I don't really think I care
if we do not get back up the SACK'd but not past the cum-ack
point... We could indicate these but I think we are talking
corner cases here...

The general mechanism, as it is currently laid out, is I
think valuable.. 

R

---

From: Randall Stewart at work <[email-masked]>
Date: Thu, 28 Jun 2001 17:51:09 -0500

> indicate whether the message can be duplicated?   And I

Yes, the context was used to handle how I would react
to the "failed data". It was the applications control
flag if you will that told ASAP what was proper handling..

> discussion. 

I definetly got off track from some of Brian's comments.
ASAP right now does NOT tell the difference between
any of the classes..

i.e. 
1) UNSENT, 
2) SENT and NOT ACKED and 
3) SACKed but no CUM-ACK past.

Reason being in the original work we did with this, which used
an old old old version of SCTP, there was NO distinction between
1 <-> 2, every thing was just tosed up... and as far as
the distinction between 2 <-> 3, the old version did not
have this distinction i.e. once it was SACK'd it could not
be revoked...

Now, I can see the value of knowing the difference betweein
1 <-> 2...

Having 3 returned seems moot to me and it could just as
likely be reported as a case 2... since it still requires
audits.. for example with the remote end..

>             What is useful and what is not useful?  And

I think only 1 and 2 are useful.. and I would classify
any messages in catagory 3, if I gave them to the
app at all as a catagory 2 as well... they still need
to be audited...

> induced failure behavior.

Not sure on Jacob's proposal... I want to think on it
more..

R

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 29 Jun 2001 01:14:07 -0500

Randall,

I suspected as much.  How does the RSERPOOL/ASAP implementation
handle the ambiguity of messages which were transmitted but not
acknolwedged by the SCTP peer and the ambiguity of messages which
were acknowledged by SCTP but not delivered to the ASAP peer?

For example, one ASAP peer is sending messages to another.  The
receiving ASAP peer dies (presumably taking the process with it and
"closing" the SCTP socket).  How do you handle the messages which
are possibly lost in the dying peer's socket receive queue when it
died?  These messages were potentially acked to the sending SCTP but
were never delivered to the ULP.  How does the sender handle
messages which were transmitted but not acked?  How is it to know
which messages were actually received by the other side (but the
acks were lost)?

How does the sender distinguish between communication loss and peer
crash?  It is possible that communication was merely lost and that
messages were in the process of being acked for which the acks were
not received due to the communication loss?  How does the sending
ASAP know which messages were received by the other side in this
case?

So of the cases following cases, hows does the sending ASAP know
which messages to send to the alternate ASAP?

(1) Unsent messages (this one's obvious)

(2) Sent but unacknowledged messages (how does the sending ASAP know
    that these messages were not received and processed by the
    peer?)

(3) Sent but gap-acked messages (how does the sending ASAP know
    whether these were processed or lost?)

(4) Sent and cum-acked messages (how does the sending ASAP know
    whether these were processed or lost?)

I don't see how the ASAP case is so different...  Could you explain
how it deals with these ambiguities without the assistance of an
application level ack?

--Brian

---

From: Randall Stewart at work <[email-masked]>
Date: Fri, 29 Jun 2001 08:30:01 -0500

Brian:

As I stated earlier.. we did not have a seperation as
we are now putting in the API. As such you either
had to retransmit all the messages or pull them
up to the application. 

The applications themselves i.e. the servers had
to do state sharing outside of rserpool...it is out
of scope of rserpool and we had several mechanisms to
do this.

It really depended on the application as to how this
was handled. Some apps wanted the data and then did
educated things on resubmitting. Others just had
ASAP resend to a peer server... the peer server
would then do its needed audits etc.

I see having the seperation as a good thing i.e. I can
now give the application client more choices...

oh.. by the way I don't think many of our apps used
application level ack's..  a couple yes.. but most
did not need it.. Of course it was not an SS7 network
we were running either :>

R