Subject: [Tsvwg] COOKIE_ECHO retransmit on different path than piggybacked DATA

From: Brad Penoff <[email-masked]>
Date: Fri, 21 Jan 2005 16:53:50 -0800 (PST)

Greetings,

I was having some very strange and intermittent behavior in my application, 
but finally I've got the perfect traces to actually see what is happening 
under the hood; as you'll see, I am unclear as to why LKSCTP is behaving 
the way it is but it may just be my misunderstanding of the protocol.  The 
traces are quite large (available upon request) but below is a full 
explanation; my wonder is if what I'm seeing is "normal".  Also, I am 
needing some clarity about ABORT's semantics, so resultingly I'm 
cross-listing.

I have this setup (on emulab):

node4 ---------lighting-fast-control-network#2------node3
   |                                                   |
   |---gatewayC----high-latency-high-noise---gatewayB--|
                     experiment-network#1

(Below I use the terms node4, n4, node 4, Node 4, etc. interchangably)

I am testing my app in a noisy situation so network #1 above is my 
primary.  For now, since I didn't fully specify which paths to bind, both 
networks are available (i.e. #2 is the secondary, where retransmissions 
normally go).

After several other things, my application hangs when nodes 3 and 4 are 
exchanging information between each other:

  n4                                    n3
----                                  ----
A - sctp_sendmsg to n3 (returns 4)    A - sctp_recvmsg (hangs)
B - sctp_recvmsg (hangs)              B - sctp_sendmsg to n4 (never
                                                                executes)

This is in the one-to-many style, and no previous association had been 
established on these sockets, so steps n4-A and n3-A effectively start 
the 4-step handshake, attempting to piggyback the data.

I have actually many nodes doing this same series of steps and I found 
that intermittently, similar "hangs" occur where the sender (n4 here) 
returns that it sent its data where the receiver (n3 here) hangs. 
Anticipating this, I was able to have a run where I got full ethereal 
traces on both nodes 3 and 4 when it hung in this place described above.

Node 4 sees the following:
frame net#   direction     type(s)
a     #1     n4 -> n3      INIT
b     #1     n3 -> n4      INIT_ACK
c     #1     n4 -> n3      COOKIE_ECHO DATA
d     #1     n4 -> n3      COOKIE_ECHO
e     #2     n4 -> n3      DATA
f     #2     n3 -> n4      ABORT
g     #1     n3 -> n4      COOKIE_ACK
h     #2     n3 -> n4      HEARTBEAT
i     #2     n4 -> n3      ABORT

Node 3 sees the following:
frame net#   direction     type(s)
a     #1     n4 -> n3      INIT
b     #1     n3 -> n4      INIT_ACK
c     #2     n4 -> n3      DATA
d     #2     n3 -> n4      ABORT
e     #1     n4 -> n3      COOKIE_ECHO
f     #1     n3 -> n4      COOKIE_ACK
g     #2     n3 -> n4      HEARTBEAT
h     #2     n4 -> n3      ABORT

All DATA's seen here on n4 and n3 have the same TSN.

<deep breath>

So, from all this, I have these questions:

1)  The piggybacked frame n4-c was apparently lost because n4-d 
is a retransmission of the COOKIE_ECHO.  n4-d is sent over network #1.  n4 
also retransmits the DATA in a separate message (n4-e), but this is over 
network #2.  Why are these retransmitted over different networks?

2)  This retransmitted DATA arrives to n3 before the COOKIE_ECHO (n3-c), 
so it is ABORT'ed (n3-d).  But just after, n3 receives n4's retransmitted 
COOKIE_ECHO (n3-e), and it is (COOKIE_)ACK'd (n3-f) even though n3 had 
already ABORT'd this association (n3-d).  Does this make sense?  Are 
ABORT's only talking about one path of an association and not the full 
association?

3)  Why isn't the DATA retransmitted after the COOKIE_ACK ?

4)  What exactly does the ABORT n4-i accomplish that n4-f didn't already 
do?

Any help you can lend would be most appreciated.  Questions 2 & 4 might be 
due to a lack of understanding on my part regarding ABORT's but I feel 
questions 1 & 3 are valid concerns for the LKSCTP implementation being 
used in a similar scenario where one path is MUCH faster than another.

Thanks,
brad

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Randall Stewart <[email-masked]>
Date: Sat, 22 Jan 2005 08:15:30 -0500

Brad Penoff wrote:
> Greetings,
> 
> I was having some very strange and intermittent behavior in my 
> application, but finally I've got the perfect traces to actually see 
> what is happening under the hood; as you'll see, I am unclear as to why 
> LKSCTP is behaving the way it is but it may just be my misunderstanding 
> of the protocol.  The traces are quite large (available upon request) 
> but below is a full explanation; my wonder is if what I'm seeing is 
> "normal".  Also, I am needing some clarity about ABORT's semantics, so 
> resultingly I'm cross-listing.
> 
> primary.  For now, since I didn't fully specify which paths to bind, 
> both networks are available (i.e. #2 is the secondary, where 
> retransmissions normally go).
> 

Brad:

Without reading further I see a bug in whatever version of
lk-sctp you are using. Note step <e> of the first set that
n4 sees. N4 is incorrectly retransmitting the DATA seperate
from the cookie-echo. Any network reordering (which occurs
in your network) would thus cause n3 to see the DATA
first, and then the cookie-echo. At the point the DATA
arrives there is no association since the COOKIE-ECHO has
not been processed... Thus it will send back an ABORT.
Next packet in to n3 <e> in the second trace establishes
the association and sends off a COOKIE-ACK... but by the
time the COOKIE-ACK arrives the ABORT that was sent prior
to this hits and knocks out the association.

Now, I would imagine, just guessing here, that you do
not have association events enabled.. otherwise you should
read (at n4) a ASSOCIATION notification that would tell you
that the assoction died .. could never be started.

The problem is that lk-sctp in this place should either:

1) When it goes to retransmit include both the COOKIE-ECHO and
    DATA in one packet (like it did originally)
<or>

2) When it goes to retransmit hold off DATA retransmissions UNTIL AFTER
    the COOKIE-ACK is received.

For KAME-SCTP we do <2> .. but either one is valid.. The one thing
that is invalid to do is what your version of the stack is doing...

> 1)  The piggybacked frame n4-c was apparently lost because n4-d is a 
> retransmission of the COOKIE_ECHO.  n4-d is sent over network #1.  n4 

Not only that why is it done at all... the retransmission of COOKIE and
DATA must be done togehter.. or data must be done AFTER cookie-ack (see
above).

> so it is ABORT'ed (n3-d).  But just after, n3 receives n4's 
> retransmitted COOKIE_ECHO (n3-e), and it is (COOKIE_)ACK'd (n3-f) even 
> though n3 had already ABORT'd this association (n3-d).  Does this make 
> sense?  Are ABORT's only talking about one path of an association and 
> not the full association?

No, it makes sense.. but its because of the bug (see above descritpion)
> 3)  Why isn't the DATA retransmitted after the COOKIE_ACK ?

Thats the only legal thing to do UNLESS you did alaternative <1>
described above.

> 4)  What exactly does the ABORT n4-i accomplish that n4-f didn't already 

This has to do with what each side thinks at the time it
sends the abort...

N4                                      N3

                         <-- ABORT since NO TCB is in place.
<---ABORT
<remove tcb>
<-------COOKIE-ACK
<ignored since we are not in COOKIE-WAIT state>
<-------Heartbeat-----------------
------ABORT----> since NO TCB in place

At the time N3 sends its C-A, it now has a TCB prior
to that it did not. At the time N4 sends its ABORT it
does not have a TCB and is processing a HB that is OOTB.

> 
> Any help you can lend would be most appreciated.  Questions 2 & 4 might 
> be due to a lack of understanding on my part regarding ABORT's but I 
> feel questions 1 & 3 are valid concerns for the LKSCTP implementation 
> being used in a similar scenario where one path is MUCH faster than 
> another.

The path's speed do bring out the bug.. but what you really
have here is a bug in lk-sctp's version you are using... if
it is the current version then Shridhar has a bug he needs
to look at...

Anyway, I hope that explanation helps..

R

-- 
Randall Stewart
803-345-0369 <or> 815-342-5222(cell)

---

From: Sridhar Samudrala <[email-masked]>
Date: Sun, 23 Jan 2005 08:07:13 -0800 (PST)

Brad,

I think Randy has clearly explained what is happening.
This could be a bug in the lksctp version you are using.
In linux 2.6.9 we have fixed a similar bug that was causing
COOKIE_ECHO and DATA to be sent on different paths of a
multihoming associations after a stale cookie error is received.
I think that fix should handle the case you are seeing. You
should see DATA bundled with COOKIE_ECHO and retransmitted on
the primary path.

Could you try your tests with linux 2.6.9 or later versions? If
you are already using a later version, then it is possible that
the above fix may not be complete.

Thanks
Sridhar

On Sat, 22 Jan 2005, Randall Stewart wrote:

>: