Subject: Re: [Tsvwg] COOKIE_ECHO retransmit on different path than piggybacked DATA

From: Brad Penoff <[email-masked]>
Date: Tue, 8 Feb 2005 17:22:33 -0800 (PST)

hey Sridhar,

I finally got my hands on some machines and recreated this locally 
between two machines running the Ubuntu 2.6.10-1-386 kernel with the 
lksctp-2.6.10-1.0.2 includes and libs.  My setup looked as it did before, 
with the primary network having high-loss and high-latency via a dummynet 
router, and the secondary network being fast.

As in the original account below, the bundled COOKIE_ECHO/DATA packet gets 
lost and each chunk gets retransmitted separately, the COOKIE_ECHO over 
the lossy network, and the DATA over the fast network.  The DATA gets 
there first causing an ABORT and an eventual application hang.

A temporary work-around for us may be to not turn on dummynet's loss until 
after the 4-step handshake.  But still this makes me nervous.  Might there 
be another work-around?

brad

On Sun, 23 Jan 2005, Sridhar Samudrala wrote:

> Brad,
>
> I think Randy has clearly explained what is happening.
> This could be a bug in the lksctp version you are using.
> In linux 2.6.9 we have fixed a similar bug that was causing
> COOKIE_ECHO and DATA to be sent on different paths of a
> multihoming associations after a stale cookie error is received.
> I think that fix should handle the case you are seeing. You
> should see DATA bundled with COOKIE_ECHO and retransmitted on
> the primary path.
>
> Could you try your tests with linux 2.6.9 or later versions? If
> you are already using a later version, then it is possible that
> the above fix may not be complete.
>
> Thanks
> Sridhar
>
> On Sat, 22 Jan 2005, Randall Stewart wrote:
>
>> Brad Penoff wrote:
>>> Greetings,
>>>
>>> I was having some very strange and intermittent behavior in my
>>> application, but finally I've got the perfect traces to actually see
>>> what is happening under the hood; as you'll see, I am unclear as to why
>>> LKSCTP is behaving the way it is but it may just be my misunderstanding
>>> of the protocol.  The traces are quite large (available upon request)
>>> but below is a full explanation; my wonder is if what I'm seeing is
>>> "normal".  Also, I am needing some clarity about ABORT's semantics, so
>>> resultingly I'm cross-listing.
>>>
>>> I have this setup (on emulab):
>>>
>>> node4 ---------lighting-fast-control-network#2------node3
>>>   |                                                   |
>>>   |---gatewayC----high-latency-high-noise---gatewayB--|
>>>                     experiment-network#1
>>>
>>> (Below I use the terms node4, n4, node 4, Node 4, etc. interchangably)
>>>
>>> I am testing my app in a noisy situation so network #1 above is my
>>> primary.  For now, since I didn't fully specify which paths to bind,
>>> both networks are available (i.e. #2 is the secondary, where
>>> retransmissions normally go).
>>>
>>> After several other things, my application hangs when nodes 3 and 4 are
>>> exchanging information between each other:
>>>
>>>  n4                                    n3
>>> ----                                  ----
>>> A - sctp_sendmsg to n3 (returns 4)    A - sctp_recvmsg (hangs)
>>> B - sctp_recvmsg (hangs)              B - sctp_sendmsg to n4 (never
>>>                                                                executes)
>>>
>>> This is in the one-to-many style, and no previous association had been
>>> established on these sockets, so steps n4-A and n3-A effectively start
>>> the 4-step handshake, attempting to piggyback the data.
>>>
>>> I have actually many nodes doing this same series of steps and I found
>>> that intermittently, similar "hangs" occur where the sender (n4 here)
>>> returns that it sent its data where the receiver (n3 here) hangs.
>>> Anticipating this, I was able to have a run where I got full ethereal
>>> traces on both nodes 3 and 4 when it hung in this place described above.
>>>
>>> Node 4 sees the following:
>>> frame net#   direction     type(s)
>>> a     #1     n4 -> n3      INIT
>>> b     #1     n3 -> n4      INIT_ACK
>>> c     #1     n4 -> n3      COOKIE_ECHO DATA
>>> d     #1     n4 -> n3      COOKIE_ECHO
>>> e     #2     n4 -> n3      DATA
>>> f     #2     n3 -> n4      ABORT
>>> g     #1     n3 -> n4      COOKIE_ACK
>>> h     #2     n3 -> n4      HEARTBEAT
>>> i     #2     n4 -> n3      ABORT
>>>
>>>
>>> Node 3 sees the following:
>>> frame net#   direction     type(s)
>>> a     #1     n4 -> n3      INIT
>>> b     #1     n3 -> n4      INIT_ACK
>>> c     #2     n4 -> n3      DATA
>>> d     #2     n3 -> n4      ABORT
>>> e     #1     n4 -> n3      COOKIE_ECHO
>>> f     #1     n3 -> n4      COOKIE_ACK
>>> g     #2     n3 -> n4      HEARTBEAT
>>> h     #2     n4 -> n3      ABORT
>>>
>>>
>>> All DATA's seen here on n4 and n3 have the same TSN.
>>
>> Brad:
>>
>> Without reading further I see a bug in whatever version of
>> lk-sctp you are using. Note step <e> of the first set that
>> n4 sees. N4 is incorrectly retransmitting the DATA seperate
>> from the cookie-echo. Any network reordering (which occurs
>> in your network) would thus cause n3 to see the DATA
>> first, and then the cookie-echo. At the point the DATA
>> arrives there is no association since the COOKIE-ECHO has
>> not been processed... Thus it will send back an ABORT.
>> Next packet in to n3 <e> in the second trace establishes
>> the association and sends off a COOKIE-ACK... but by the
>> time the COOKIE-ACK arrives the ABORT that was sent prior
>> to this hits and knocks out the association.
>>
>> Now, I would imagine, just guessing here, that you do
>> not have association events enabled.. otherwise you should
>> read (at n4) a ASSOCIATION notification that would tell you
>> that the assoction died .. could never be started.
>>
>> The problem is that lk-sctp in this place should either:
>>
>> 1) When it goes to retransmit include both the COOKIE-ECHO and
>>     DATA in one packet (like it did originally)
>> <or>
>>
>> 2) When it goes to retransmit hold off DATA retransmissions UNTIL AFTER
>>     the COOKIE-ACK is received.
>>
>> For KAME-SCTP we do <2> .. but either one is valid.. The one thing
>> that is invalid to do is what your version of the stack is doing...
>>
>>
>>>
>>> <deep breath>
>>>
>>> So, from all this, I have these questions:
>>>
>>> 1)  The piggybacked frame n4-c was apparently lost because n4-d is a
>>> retransmission of the COOKIE_ECHO.  n4-d is sent over network #1.  n4
>>> also retransmits the DATA in a separate message (n4-e), but this is over
>>> network #2.  Why are these retransmitted over different networks?
>>
>> Not only that why is it done at all... the retransmission of COOKIE and
>> DATA must be done togehter.. or data must be done AFTER cookie-ack (see
>> above).
>>
>>>
>>> 2)  This retransmitted DATA arrives to n3 before the COOKIE_ECHO (n3-c),
>>> so it is ABORT'ed (n3-d).  But just after, n3 receives n4's
>>> retransmitted COOKIE_ECHO (n3-e), and it is (COOKIE_)ACK'd (n3-f) even
>>> though n3 had already ABORT'd this association (n3-d).  Does this make
>>> sense?  Are ABORT's only talking about one path of an association and
>>> not the full association?
>>
>> No, it makes sense.. but its because of the bug (see above descritpion)
>>>
>>> 3)  Why isn't the DATA retransmitted after the COOKIE_ACK ?
>>
>> Thats the only legal thing to do UNLESS you did alaternative <1>
>> described above.
>>
>>>
>>> 4)  What exactly does the ABORT n4-i accomplish that n4-f didn't already
>>> do?
>>>
>>
>> This has to do with what each side thinks at the time it
>> sends the abort...
>>
>>
>> N4                                      N3
>>
>>                          <-- ABORT since NO TCB is in place.
>> <---ABORT
>> <remove tcb>
>> <-------COOKIE-ACK
>> <ignored since we are not in COOKIE-WAIT state>
>> <-------Heartbeat-----------------
>> ------ABORT----> since NO TCB in place
>>
>> At the time N3 sends its C-A, it now has a TCB prior
>> to that it did not. At the time N4 sends its ABORT it
>> does not have a TCB and is processing a HB that is OOTB.
>>
>>
>>
>>>
>>> Any help you can lend would be most appreciated.  Questions 2 & 4 might
>>> be due to a lack of understanding on my part regarding ABORT's but I
>>> feel questions 1 & 3 are valid concerns for the LKSCTP implementation
>>> being used in a similar scenario where one path is MUCH faster than
>>> another.
>>
>> The path's speed do bring out the bug.. but what you really
>> have here is a bug in lk-sctp's version you are using... if
>> it is the current version then Shridhar has a bug he needs
>> to look at...
>>
>> Anyway, I hope that explanation helps..
>>
>> R
>>
>>>
>>> Thanks,
>>> brad
>>>
>>> _______________________________________________
>>> tsvwg mailing list
>>> [email-masked]
>>> https://www1.ietf.org/mailman/listinfo/tsvwg
>>>
>>>
>>
>>
>> --
>> Randall Stewart
>> 803-345-0369 <or> 815-342-5222(cell)
>>
>> _______________________________________________
>> tsvwg mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/tsvwg
>> :
>

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Tue, 8 Feb 2005 23:01:15 -0700

Brad,

On Tue, 08 Feb 2005, Brad Penoff wrote:

--X-snip-X--
> after the 4-step handshake.  But still this makes me nervous.  Might there 

The following strikes me as possibilities: ;)

1.   Change the code.
2.   Use a different stack.
3.   Report a bug.

None of which are of much interest on TSV.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦

---

From: Michael Tuexen <[email-masked]>
Date: Wed, 9 Feb 2005 11:28:37 +0100

Dear all,

you must either
- bundle the COOKIE-ECHO and DATA at retransmission also
or
- delay the retransmission of the DATA until the reception
   of the COOKIE-ACK.

Best regards
Michael

On Feb 9, 2005, at 2:22 Uhr, Brad Penoff wrote:

> lksctp-2.6.10-1.0.2 includes and libs.  My setup looked as it did 
> before, with the primary network having high-loss and high-latency via 
> a dummynet router, and the secondary network being fast.
>
> As in the original account below, the bundled COOKIE_ECHO/DATA packet 
> gets lost and each chunk gets retransmitted separately, the 
> COOKIE_ECHO over the lossy network, and the DATA over the fast 
> network.  The DATA gets there first causing an ABORT and an eventual 
> application hang.
>
> A temporary work-around for us may be to not turn on dummynet's loss 
> until after the 4-step handshake.  But still this makes me nervous.  
> Might there be another work-around?
>
> brad
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real 
> users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> Lksctp-developers mailing list
> [email-masked]
> https://lists.sourceforge.net/lists/listinfo/lksctp-developers
>