Subject: Section 9: Heartbeat Messages

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 7 Sep 2005 13:57:28 -0400

Folks,

At the moment we are ambiguous about the heartbeat; the draft states
as it being either from CE->FE or FE->CE and in-fact doesn't discount
CE<->CE or FE<->FE (which is outside the scope).

a) What we have found is that the most effective way to do this is to
actually have heartbeats going from CE->FE since the CE is the master,
mostly. More importantly is that this is allows for more complexity on
the CE than on the FE (which is important). I also like the fact that 
if we ever use a multicast capable TML, we can optimize further the
amount of traffic on the wire - which is impossible if direction was
FE->CE.

b) We have implemented the thought originated from Joel to have the
heartbeats being optimized (for scalability purposes) to be adaptively
traffic-sensitive. In such a scheme heartbeats are only sent if no other
traffic has been sent to the FE since the last heartbeat interval. This
is a good optimization since it means that in idle (CE->FE) periods a
lot more heartbeats are sent than in busy periods.

c) Further observation is that the directionality is important. If
CE->FE is the direction, all this is doing is health checking from the
FEs perspective of the CE. Vice-versa if direction was FE->CE.
Ideally we would like both health checks. So one thing we are
experimenting with is an extension of #b above. 
Instead of just the check for whether we have sent any traffic to the FE
in the last heartbeat interval, we also check if we have received any.
In most cases if we sent, we probably also received a response, so the
check doesnt make a difference. In the cases where we have not received
traffic from the FE, we set the ACK flag which tells the FE to actually
respond to the HB. Only in this case do we have bidirectional HB, i.e, a
ping-pong.

Could people please express their opinions on this? interop will be
affected if any one implementation doesnt obey _all_ 3 rules above.
I think all 3 above are very useful and if theres agreement i will edit
the draft to reflect things.
The FE Protocol Object LFB will have to change, but lets not go there
yet - lets agree on the above first.

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Thu, 8 Sep 2005 21:02:19 +0800

Jamal,

I think of the heartbeat mechanism a bit differently, summarized as below:

1. a heartbeat acts as a signal to show the liveness of the sender, therefore, I cannot see the value to have a heartbeat response. What does such response actually mean? 

2. to have a piggy-backing heartbeat is valuable. We have already had issue 11 for this, and I think it has been accepted. The only thing left is we have not had it reflected in the draft text. It's really not a big change, and I suppose the following text to incorparate the piggy-backing heartbeat:

<original P.42>
FE Protocol attributes (can be read and set):
...
      *  Heartbeat Interval.  Defualt Value = 300 msec
</original P.42>

We may add the piggy-backing heartbeat by modifying this attribute as something like:

<change to>
FE Protocol attributes (can be read and set):
...
      * Heartbeat policy and parameters, which includes fields as:

        . Heartbeat policy, which further includes two fields:

           CE heartbeat policy: to tell FE which heartbeat policy the CE chooses.

            0 - CE no heartbeat
            1 - CE normal heartbeat
            2 - CE piggy-backing heartbeat

           FE heartbeat policy: to command the FE to use the heartbeat policy.

            0 - FE no heartbeat
            1 - FE normal heartbeat
            2 - FE piggy-backing heartbeat

        . CE heartbeat interval, to tell FE the heartbeat interval CE uses. if not set or set to 0, it means CE chooses a default value, which is 300msec

        . FE heartbeat interval, to command FE to use the interval for the FE heartbeat. If not set or set to 0, it means FE must choose the default value, which is 300msec

</change to>

Note in above, CE need to tell FE how it generates heartbeat also, so that FE can correctly sense the CE liveness.

We may also need to add some text in the Heartbeat Message section mentioning that "the heartbeat is generated according to the heartbeat policy set in the FE Protocol LFB attribute, which specifies whether the heartbeat generated is normal or piggy-backing over other messages."

Any thoughts?

Thanks,
Weiming

----- Original Message ----- 
From: "Jamal Hadi Salim" <[email-masked]>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 8 Sep 2005 09:30:27 -0400

Weiming,

On Thu, 2005-08-09 at 21:02 +0800, Weiming Wang wrote:
> 1. a heartbeat acts as a signal to show the liveness of the sender, 

Yes, this is the way i see too ;-> More below.

> therefore, 
> I cannot see the value to have a heartbeat response. What does such response 
> actually mean? 
> 

CE->FE is announcing liveliness of CE to FE.
A response from FE is to announce liveliness of FE to CE.
Given Joels optimization - the only time we have traffic both directions
is when neither the CE nor FE has any activity. There could be some
semi-quiet period, for example it is possible to have FE send
synchronous/periodic events like stats in which case we know it is
alive.  
In most cases the interaction will be config/query form.
In the case of config and query from CE->FE we know the FE is alive when
we receive a response for the query/config.
So the optimization works very well, only we need to use it in both
directions to reduce the number of HB messages in busy periods.

> 2. to have a piggy-backing heartbeat is valuable. We have already had issue 
> 11 for this, and I think it has been accepted. The only thing left is we have 
> not had it reflected in the draft text. It's really not a big change, and I 
> suppose the following text to incorparate the piggy-backing heartbeat:
> 

Ok, so issue 11 is one of the reasons that prompted me to post actually.
BTW, it should be open until text is entered.

I was a big fan of piggybacking before - After we implemented current HB
i found it to be unnecessary to have piggybacking.
This is because _every message_ between a CE<->FE is a liveliness
checker. Does this make sense? I am not asking for its removal if people
have implemented and found it necessary (we too in the past have
implemented that way).

> We may add the piggy-backing heartbeat by modifying this attribute as something like:

[..]

> correctly sense the CE liveness.

Refer to my comments above. I think it is not necessary for the FE to
send any heartbeats at all. It will always be responding to queries and
configs and from that we can tell its liveliness (given the
optimizations). 
My recommendation is that since the CE is the master it should be the
one issuing the ACK flag when it hasnt heard from the FE when the
heartbeat interval expires.

Thoughts?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Thu, 8 Sep 2005 10:08:02 -0400

There are two (at least) valid implementation models for heartbeats...
One can say that the heartbeat is a signal that the sender is live.  Then 
each side independently sends heartbeats.
One can also say that heartbeat is a connection liveness monitor, managed 
by one side.  This usually requires the other side to respond.  This has 
the advantage or disadvantage that the behavior is driven by one side.

Yours,
Joel

At 09:02 AM 9/8/2005, Weiming Wang wrote:
>therefore, I cannot see the value to have a heartbeat response. What does 
>such response actually mean?
>
>2. to have a piggy-backing heartbeat is valuable. We have already had 
>issue 11 for this, and I think it has been accepted. The only thing left 
>is we have not had it reflected in the draft text. It's really not a big 
>change, and I suppose the following text to incorparate the piggy-backing 
>heartbeat:
>
><original P.42>
>         . CE heartbeat interval, to tell FE the heartbeat interval CE 
> uses. if not set or set to 0, it means CE chooses a default value, which 
> is 300msec
>
>         . FE heartbeat interval, to command FE to use the interval for 
> the FE heartbeat. If not set or set to 0, it means FE must choose the 
> default value, which is 300msec
>
></change to>
>
>We may also need to add some text in the Heartbeat Message section 
>mentioning that "the heartbeat is generated according to the heartbeat 
>policy set in the FE Protocol LFB attribute, which specifies whether the 
>heartbeat generated is normal or piggy-backing over other messages."
>
>Any thoughts?
>
>Thanks,
>Weiming
>

---

From: Weiming Wang <[email-masked]>
Date: Thu, 8 Sep 2005 22:25:58 +0800

jamal,

Let's first syncronize something. My understanding of piggy-backing is not to let a heartbeat message piggy-backed on other messages, instead, it is exactly the meaning that any messages can act for heartbeat. With this in mind, I think we are on the same page with the use of such piggy-backing (maybe need some text to clarify it more in the text that I proposed). 

Now the only thing left is, you seem to prefer to use the CE->FE heartbeat as some king of liveness strobe message, i.e., CE sends a heartbeat to FE only in the hope of getting a response so as to sense the liveness of the FE. We may have two shortcuts using such scheme, I suppose:

1. it's still not able to let FE to sense the liveness of CE (we may go to the discussion on if FE need to sense liveness of CE, but let's suppose it is needed). Let's try to give a scenario for this: say CE is quite, and FE is frequently sending events.  Because CE can sense the FE, CE will also not send any CE->FE strobe message then. In this case, CE can sense FE liveness, while FE cann't sense CE.

2. I'm not sure if it's more optimized to sense FE by CE actively sending strobe message than by the FE actively sending heartbeat message, because the former needs two messages, while the latter only one.

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 8 Sep 2005 10:32:03 -0400

On Thu, 2005-08-09 at 10:08 -0400, Joel M. Halpern wrote:
> by one side.

The connection liveliness aspect i think is the responsibility of the
TML. As an example a simple KEEPALIVE on the TCP TML may suffice in most
cases. Note TCP KEEPALIVEs get acks back and are issued by one side. I
think this distinction should be noted in the draft.

>   This usually requires the other side to respond.  This has 

I claimed in my other email it is an advantage for the CE to be the
driver - not just because the relationship is one between a master (CE)
and a slave (FE) but rather because it really simplifies things on the
FE side.  I think the fact that there is a one-to-many relation between
the CE and FEs implies, depending on the TML and FEID mapping, that a
further optimization to be used is sending a single multicast message on
the wire (except in the case where responses are needed).

Thoughts?

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 8 Sep 2005 11:03:30 -0400

Weiming,

Thanks for keep-aliving the discussion, and now for my ACK
below .. ;->

On Thu, 2005-08-09 at 22:25 +0800, Weiming Wang wrote:
> jamal,
> 
> Let's first syncronize something. My understanding of 
> piggy-backing is not to let a heartbeat message piggy-backed 
> on other messages, instead, it is exactly the meaning that any 
> messages can act for heartbeat. 

Ok, my understanding was actually the opposite of yours from reading
issue 11;-> i.e a heartbeat is sent and it includes some other content
like stats.
 
> With this in mind, I think we are 
> on the same page with the use of such piggy-backing (maybe need 
> some text to clarify it more in the text that I proposed). 
> 

We are in sync on this then. My only doubts at this point is whether
we should call the optimization piggy-backing or traffic-sensitive
heartbeating ;-> And whether we need to have the flags you mentioned
earlier.

> Now the only thing left is, you seem to prefer to use the 
> CE->FE heartbeat as some king of liveness strobe message, 
> i.e., CE 
> sends a heartbeat to FE only in the hope of getting a response so 
> as to sense the liveness of the FE. We may have two shortcuts 
> using such scheme, I suppose:
> 
> 1. it's still not able to let FE to sense the liveness of CE (we 
> may go to the discussion on if FE need to sense liveness of CE, 
> but let's suppose it is needed). 

The FE has HA information on multiple CEs  - so my take is that it may
need to know the liveliness details so it can failover to one of the
other CEs listed.

> Let's try to give a scenario for 
> this: say CE is quite, and FE is frequently sending events.  
> Because CE can sense the FE, CE will also not send any CE->FE 
> strobe message then. In this case, CE can sense FE liveness, while 
> FE cann't sense CE.
> 

I think i mispoke in my last email. The scheme works based on direction
of sending. i.e.
CE->FE is sent if CE hasnt sent in the last interval.
FE->CE is requested via ACK if during that same period, the CE saw no
message from the FE. 

Does this make sense?

> 2. I'm not sure if it's more optimized to sense FE by CE actively 
> sending strobe message than by the FE actively sending heartbeat 
> message, because the former needs two messages, while the latter 
> only one.

Unless i misunderstood you, the two are equivalent in the number of
messages.

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Thu, 8 Sep 2005 23:46:59 +0800

> heartbeating ;-> And whether we need to have the flags you mentioned
Let's differ the discussion on whether we call it heartbeat or not till we see the solution :)
For the flags, my thought is we may need to keep the flexibility as no such heartbeat or with normal heartbeat, but we can also differ this I think.

> CE->FE is sent if CE hasnt sent in the last interval.
Ok, we syncronize more then.

> FE->CE is requested via ACK if during that same period, the CE saw no
Now, let's see the difference then. Seems this is also the only differnece left.

My thought is, for FE->CE, it has nothing to do with CE->FE anything. FE only sends heartbeat(let's still call it this way) when the interval ends.

May I rewrite your thought as: if former CE->FE heartbeat is set ACK and if the interval ends, then the FE needs to send back a heartbeat response to CE, or else it will not send anything extra.

This still leads to some shorts, I'm afraid:
1. this means for some CE->FE heartbeat, even set with ACK, it may not get response (because some FE->CE message overrides it)
2. What then if the former CE->FE heartbeat has not set the ACK?
3. What is the real value to use ACK and use heartbeat response, rather than to use just a FE->CE heartbeat then?

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 8 Sep 2005 12:06:42 -0400

> Let's differ the discussion on whether we call it heartbeat or not till 

NP.

> Now, let's see the difference then. Seems this is also the only difference left.
> 
> My thought is, for FE->CE, it has nothing to do with CE->FE anything. FE only 
> sends heartbeat(let's still call it this way) when the interval ends.
> 
> May I rewrite your thought as: if former CE->FE heartbeat is set ACK and if 
> the interval ends, then the FE needs to send back a heartbeat response to CE, 
> or else it will not send anything extra.
> 
> 1. this means for some CE->FE heartbeat, even set with ACK, it may not get 
> response (because some FE->CE message overrides it)

This is fine if you assume any message from the FE is a response to the
HB sent by CE. i.e it cancels need to wait for response.

> to use just a FE->CE heartbeat then?

Lets enumerate all scenarios (luckily theres only 4) in which something
may be idle or busy during HB interval. When timer expires:

case 00: 
CE idle, FE idle 
--> CE Send HB with no ACK

case 01: 
CE idle, FE active
---> Send HB with ACK. 
-----> Wait for response, if _any_ msg from FE, we claim FE is alive

Case 10:
CE active, FE idle
--> same as case 01. I think this is the case you probably said there
was two messages instead of one.

case 11:
CE active, FE active
--> Do nothing

Ok, lets use the above scenarios for further discussion.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Thu, 8 Sep 2005 12:10:35 -0400

> -----> Wait for response, if _any_ msg from FE, we claim FE is alive

Sorry, 00 CE Send HB with ACK and 01 with no ACK.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 9 Sep 2005 20:53:56 +0800

Now, I can understand your thought well. I think the only difference may lie in where the timer for intervals is put. The scheme you proposed also needs two timers, only both timers all reside in CE, one is for CE to measure the interval, one is to measre the FE interval. Two cases will trigger the CE to send HB, when CE itself arrives message sending interval, and when CE does not receive FE message within an interval. 

I'm just a little worry that for the latter timer, CE may burden quite some, as we know, we may have hundreds of FEs, which may make CE quite hard to manage.

Thanks,
Weiming

---

From: Robert Haas <[email-masked]>
Date: Mon, 12 Sep 2005 16:45:16 +0200

Weiming, Jamal,
A few comments on the HB issue:
- the HB is a mechanism to check liveliness of the PL layer. If the PL 
crashes at the FE or the CE, then (even if the TML is still working) the 
association must go down.
- somehow the FE has to know when the association with its CE goes down, 
because it must then decide whether to continue forwarding packets or 
not, among other things. So HBs from CEs to FEs are obviously necessary.
- the CE would like to know the same thing, so it can perform HA 
procedures, for instance. So liveliness indicators from the FEs are 
necessary.
- this can be achieved with HB messages, or any other PL message 
whenever they are exchanged. BTW, I think the HB piggy-backing was 
proposed by Evangelos Haleplidis some time ago. And the idea was a 
"virtual piggy-backing", i.e., nothing else but a normal PL message is 
exchanged.
- multicasting HBs from the CE to all FEs is a good idea. But HBs (or 
ACKs to HBs) from the FEs to the CE do not benefit much from multicast.

So the two solutions how to perform PL-liveliness checks are:
1) "CE driven": The FE does not originate any HB by itself, it merely 
responds to CE HBs that have the ACK flag ON.
2) "independent" The FE generates its own HBs according to a timer value 
that the CE can set.

I find solution 2) simpler to implement. Note that both solutions can 
benefit from "virtual piggy-backing". The same amount of messages is 
necessary in both cases (assuming the HB period of the FE and CE are the 
same).

Regards,
-Robert

-- 
Dr. Robert R. Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-44-724-8698  fax +41-44-724-8578  http://www.zurich.ibm.com/~rha

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 12 Sep 2005 10:55:47 -0400

Hi Weiming,

I think what you are referring to is more timer implementation issues.
As an example in the Linux kernel i have tested upto 100K different
timers (one per flow) with no issues. There are many ways to achieve
this goal and well known algorithms exist - If you want we can talk
offline further on this; i just dont see it as an issue that will
overburden the CE. 

cheers,
jamal

On Fri, 2005-09-09 at 20:53 +0800, Wang,Weiming wrote:
> Now, I can understand your thought well. I think the only difference 
> may lie in where the timer for intervals is put. The scheme you 
> proposed also needs two timers, only both timers all reside in CE, 
> one is for CE to measure the interval, one is to measre the FE interval. 
> Two cases will trigger the CE to send HB, when CE itself arrives message 
> sending interval, and when CE does not receive FE message within an interval. 
> 
> I'm just a little worry that for the latter timer, CE may burden quite some, 
> as we know, we may have hundreds of FEs, which may make CE quite hard to manage.
> 
> Thanks,
> Weiming
> 
> > 
> > --> CE Send HB with ACK
> > ---> Send HB with no ACK.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 12 Sep 2005 11:08:30 -0400

Robert,

> association must go down.

Indeed. Infact one could go further than this and check the health of
some pieces of the PL before sending a HB - but i would leave that to
implementation details.

> not, among other things. So HBs from CEs to FEs are obviously necessary.

yes

> procedures, for instance. So liveliness indicators from the FEs are 

yes

> "virtual piggy-backing", i.e., nothing else but a normal PL message is 

Ok, thats what we are settling on.

> ACKs to HBs) from the FEs to the CE do not benefit much from multicast.

Let us leave this to be a TML dependent discussion (to avoid
distraction). 
To comment though: with multicast one way you would benefit from
cutting down the traffic by O(n) where n is the numberof(FEs -1)

> necessary in both cases (assuming the HB period of the FE and CE are the 

The FE should still have a timer - a dead interval timer which is
configured on FE Protocol object by the CE. This is not a heartbeat
timer. On expiry of this timer, if the Fe hasnt heard from the CE (via
heartbeats or piggybacks) it declares the association dead and invokes
its HA rules. So it is still #1 i see with no need for #2 (i.e HB from
FE->CE).

cheers,
jamal

---

From: Avri Doria <[email-masked]>
Date: Tue, 13 Sep 2005 11:17:01 +0200

On 12 sep 2005, at 16.45, Robert Haas wrote:

> "virtual piggy-backing", i.e., nothing else but a normal PL message  
> is exchanged.

if i understand this correctly, it means that the receiver has to  
treat the PL message as if it were a HB and take the appropriate  
actions, e.g. resetting timers.  Does it do this with all PL messages  
or just come?  My question is wether this doesn't become more of a  
burden then actually just having a specialized HB message going back  
and forth.

> - multicasting HBs from the CE to all FEs is a good idea. But HBs  
> (or ACKs to HBs) from the FEs to the CE do not benefit much from  
> multicast.

not unless there is more the one CE tracking liveliness.

> 1) "CE driven": The FE does not originate any HB by itself, it  
> merely responds to CE HBs that have the ACK flag ON.
> 2) "independent" The FE generates its own HBs according to a timer  
> value that the CE can set.
>
> I find solution 2) simpler to implement. Note that both solutions  
> can benefit from "virtual piggy-backing". The same amount of  
> messages is necessary in both cases (assuming the HB period of the  
> FE and CE are the same).

i also recommend solution 2.

once thing the CE has to be careful about, especially in the case of  
massive FE systems is to make sure the HB are not synchronized to  
each other.

a.

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 13 Sep 2005 11:52:49 -0400

Ok, folks we need to close on this issue so we can progress on other
unresolved issues.
More below.

> burden then actually just having a specialized HB message going back  

I think it would be implementation specific. For example a PL message
could set a boolean state to indicate some traffic was seen which is
then cleared by the expired HB timer etc.

> not unless there is more the one CE tracking liveliness.

Lets leave this out of the current discussion - it would be TML
specific.

> i also recommend solution 2.

How about this as a middle ground so we can close this:

- the CE can instruct the FE to send its own heartbeats at defined
intervals. It should also define its dead interval. All these attributes
will reside in the FE Proto Object. 

- The CE MUST be able to instruct the FE to send heartbeats.
- The CE MAY decide not to instruct the FE to send heartbeats and
instead drive them via ACKs.
- The FE MAY support sending heartbeats. Which means we need a
capability announcement to indicate the FE can do this via FE Protocol
Object.

This way one can choose to never implement heartbeats from the FE (I
know this will be our preference for reasons i described earlier based
on implementation experience) but the CE should still deal with them
when it has to.

On a personal level i see complexity supporting both but if thats what
it takes to reach consensus so be it. 

> massive FE systems is to make sure the HB are not synchronized to  

Very good point. I will incorporate in the text.

cheers,
jamal

---

From: Robert Haas <[email-masked]>
Date: Wed, 14 Sep 2005 16:35:05 +0200

Jamal,

>>burden then actually just having a specialized HB message going back  
We can make this an optional feature of the FE: if the CE wishes so, the 
FE will refrain from sending HBs in case it is already sending other PL 
messages.

>[...]
By default, I suggest that the FE must send HBs, without the virtual 
piggy-back feature. The CE can change this if the FE allows it: for 
instance, enable HB piggy-backing, or simply turn off HBs. We should 
decide these two features (HB piggybacking and turning off HB) should be 
mandatory or optional features of the FE. I propose that they are mandatory.

Regards,

---

From: Weiming Wang <[email-masked]>
Date: Thu, 15 Sep 2005 00:34:24 +0800

Robert and Jamal,

Jamal, not sure why your post of this did not show on the list. 

I still can not see the much value to have CE be always active for HBs even from FE. I'v already summarized the shorts for this as:

1. It makes CE have to manage time intervals for even hundreds of FEs. Although Jamal mentioned that there is a good algorithm for such management to make one timer for all such interval notification, I just think even a log for so many FEs is a big extra burden for CE, which I think can be removed and distributed to individual FEs.

2. It makes us have to define one more message - the HB response message, though it may not be much difficult.

3. ACK on and off is also an extra work for customer to use, which can also be removed.

On the otherhand, I can see the only benifit be that may bring is it may make CE more like a master.

Jamal, can you present some more benifits so that we can see the value more?

Thanks,
Weiming

----- Original Message ----- 
From: "Robert Haas" <[email-masked]>
To: <[email-masked]>
Sent: Wednesday, September 14, 2005 10:35 PM
Subject: Re: Closure needed please comment WAS(Re: Section 9: Heartbeat Messages
[Weiming] 

Regards,

-- 
Dr. Robert R. Haas

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 14 Sep 2005 12:47:02 -0400

> mandatory or optional features of the FE. I propose that they are mandatory.

The above would be fine with me.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 14 Sep 2005 13:00:33 -0400

> Jamal, not sure why your post of this did not show on the list. 

Hrm, that is strange. I saw it echo back and it seems at least Robert
saw it as well. This is one of the reason i always CC the person who
sent the email.

> I still can not see the much value to have CE be always active for
>  HBs even from FE. I'v already summarized the shorts for this as:
> 
> 1. It makes CE have to manage time intervals for even hundreds of 
> FEs. Although Jamal mentioned that there is a good algorithm for 
> such management to make one timer for all such interval notification, 
> I just think even a log for so many FEs is a big extra burden for CE, 
> which I think can be removed and distributed to individual FEs.
> 

On the scaling issues in this particular case, I am going to disagree:
The piggybacking scheme allows HBs to be a function of how busy the
connection is between CE-FE. HB will be sent more only when the system
is idle. Note, however:

I am not proposing disallowing HB from the FE to CE - my suggestion is
to make that an option; if the NE setup is to do it that way it should
still work.

> 2. It makes us have to define one more message - the HB response 
> message, though it may not be much difficult.
> 

Why do you need a new message?
A HB with an ACK is responded to with a HB with no ACK

> 3. ACK on and off is also an extra work for customer to use, which 

There should be no ACK configuration. The behavior in general may be
configured with a default of some sort. i.e the default could be to have
the FE send HBs if it can do that. If it cant do that, then they are
driven by CE etc.

> it may make CE more like a master.

Yes, that is beneficial - but probably not strong enough on its own.
I claim that the major benefit is simplifying the FE side.

> Jamal, can you present some more benifits so that we can see the 

see above.

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 16 Sep 2005 12:02:30 +0800

Hi, 

Here is the text proposed for the HB. Pls comments.

For FE capability for heartbeats:

<original>
 o  FE Protocol capabilities (read-only):

      *  Supported ForCES protocol version(s) by the FE

      *  Supported ForCES FE model(s) by the FE

      *  Some TML capability description(s)
</original>

<changeto>
 o  FE Protocol capabilities (read-only):

      *  Supported ForCES protocol version(s) by the FE

      *  Supported ForCES FE model(s) by the FE

      *  Some TML capability description(s)
    
      *  FE capability to generate Heatbeats: Capable or Incapable. 
</changeto>

For FE attributes for heartbeats:

<original P.42>

<change to>

            0 - CE no heartbeat. In this case, CE does not send any Heartbeat messages to FE. 
            1 - CE normal heartbeat. In this case, CE sends Heartbeat messages to FE at the time intervals specified by the CE heartebeat interval parameter as below.
            2 - CE piggy-backing heartbeat. In this case, CE sends Heartbeat messages to FE only when CE reaches the time interval with no other message to be sent in the interval.
            3 - CE piggy-backing heartbeat with FE heartbeat request. In this case, besides CE sends piggy-backing heartbeat, CE also sends Heartbeat request message (also a normal HB message, only whose ACK flag being set to AlwaysACK) to FE when CE has not received any message from FE for the interval period. When this CE heartbeat policy is choosen, the FE heartbeat policy below should be set to 'FE no heartbeat'.

            0 - FE no heartbeat. 
            1 - FE normal heartbeat.
            2 - FE piggy-backing heartbeat. In this case, FE sends heartbeat messages to FE only when it reaches the time interval of no message sending.

</change to>

---

From: Avri Doria <[email-masked]>
Date: Fri, 16 Sep 2005 14:25:42 +0200

Hi,

I think it is essentially fine except that i think calling it piggy  
back is misleading, i.e. sounds like there is a HB included in the  
message.  I would recommend implicit or something like that.

a.

On 16 sep 2005, at 06.02, Wang,Weiming wrote:

>             1 - CE normal heartbeat. In this case, CE sends  
> Heartbeat messages to FE at the time intervals specified by the CE  
> heartebeat interval parameter as below.
>             2 - CE piggy-backing heartbeat. In this case, CE sends  
> Heartbeat messages to FE only when CE reaches the time interval  
> with no other message to be sent in the interval.
>             3 - CE piggy-backing heartbeat with FE heartbeat  
> request. In this case, besides CE sends piggy-backing heartbeat, CE  
> also sends Heartbeat request message (also a normal HB message,  
> only whose ACK flag being set to AlwaysACK) to FE when CE has not  
> received any message from FE for the interval period. When this CE  
> heartbeat policy is choosen, the FE heartbeat policy below should  
> be set to 'FE no heartbeat'.
>
>             2 - FE piggy-backing heartbeat. In this case, FE sends  
> heartbeat messages to FE only when it reaches the time interval of  
> no message sending.
>
>
>         . CE heartbeat interval, to tell FE the heartbeat interval  
> CE uses. if not set or set to 0, it means CE chooses a default  
> value, which is 300msec
>
>         . FE heartbeat interval, to command FE to use the interval  
> for the FE heartbeat. If not set or set to 0, it means FE must  
> choose the default value, which is 300msec
>
> </change to>
>
>

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 16 Sep 2005 09:48:07 -0400

Unfortunately I didn't see Weiming's original email.

Can we make the piggybacking aspect mandatory? I dislike that name too.
I think we all agree it is a _very useful_ feature. Is there _any_
reason why you would not want that feature on?

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Fri, 16 Sep 2005 21:56:19 +0800

Jamal, 

I'm not sure if you mean the normal HB is helpless. I agree if no other objections. Whereas the case 'no heartbeat' should be kept. 

After we agree no normal HB case, then one way to avoid the word piggybacking is, we just use it as heartbeat. That means, in any cases when we use heartbeat, it always follows the rule of piggybacking.

Is it all right?

Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 16 Sep 2005 10:04:04 -0400

On Fri, 2005-16-09 at 21:56 +0800, Wang,Weiming wrote:
> Jamal, 
> 
> I'm not sure if you mean the normal HB is helpless. I agree 
> if no other objections. Whereas the case 'no heartbeat' 
> should be kept. 
> 
> After we agree no normal HB case, then one way to avoid 
> the word piggybacking is, we just use it as heartbeat. That 
> means, in any cases when we use heartbeat, it always follows 
> the rule of piggybacking.
> 
> Is it all right?
> 

I think i agree;-> Maybe we can settle on the concal. In other words
what you are saying -if i understood correctly - is it doesnt matter
what message goes out from the FE it is a polymorphic heartbeat.
If this is what you are saying i agree with you.

I was only worried about the extra knob to turn on piggy-backing in
FEPO. If we make it mandatory to do the baove then we can remove that
knob.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 16 Sep 2005 11:37:45 -0400

To clarify so we can close.
Someone please bounce this message to Weiming as well incase he doesnt
see it. I see to have problems sending emails to him sometimes.

> >       *  FE capability to generate Heatbeats: Capable or Incapable.

As per concal, we are now saying the FE MUST implement heartbeats but CE
can turn the generation of heartbeats off. Which means we dont need
capability above.

> > be set to 'FE no heartbeat'.

If we say the CE MUST send heartbeats then we dont need CE heartbeat
policy and we can get rid of above "knobs" altogether.

> > heartbeat messages to FE only when it reaches the time interval of  

If we agree that associating liveliness of the CE<->FE with heartbeats
(what is referred to above as piggy-backing) is always going to be
implemented i.e a MUST then 12 only need option 0 or 1.

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Tue, 20 Sep 2005 10:45:27 +0800

Jamal,

I'm sorry to reply a bit late. I'm actually trying to understand your meaning of FE MUST and CE MUST for heartbeat generating in your response. I just found it is inconsistent in some way, therefore hard to follow your meaning. I'm not sure if you actually mean "FE MUST generate HBs" instead of "CE MUST". My general comment is, the scheme for a "no heartbeat" of either CE or FE should be allowed, so as to support a very confident system, where implementors believe they don't need the HBs at all for the liveness check.

I see the main difference lying between us is, if you now agree to allow the HB scheme as the FE automatically generates HB messages rather than just resonse to CE HB messages? 

I can actually agree to let the scheme you proposed as a defualt scheme.

Based on your idea and my thoughts above, I try to summarize the text again as below, pls just comments more:

<change to>

  . CE heartbeat policy - This policy actually tells the FE how to check the CE liveness. to tell FE which heartbeat policy the CE chooses, so that FE can check the CE liveness based on this policy. The values are:

       0(default) - This policy specifies that CE will send a Heartbeat Message to FE whenever CE reaches a time interval within which no any other messages were sent from CE to FE. Associated with this policy is a 'CE heartbeat dead interval' parameter as described below. FE check the CE liveness just by checking if it has or has not received any message (including Heartbeat Message) for the 'CE heartbeat dead interval' period.

       1 - This policy specifies that CE will not send any Heartbeat Message to FE in any case. This actually means CE does not want to ask FE to check the CE liveness. 

       Others - reserved.

   . FE heartbeat policy - This policy tells FE how it should act so that CE can check its liveness based on the actions. The values are:

       0(default) - This policy actually specifies no more action for FE heartbeat generation. To let CE check its liveness, the FE may only just make responses to CE when CE sends Heartbeat Request Message (also a normal HB message, only with whose ACK flag is set to AlwaysACK) with a Heartbeat Response Message (also a normal HB message, only with whose ACK flag is set to NoACK). 

       1 - This policy specifies that FE must actively send a Heartbeat Message if it reaches a time interval, within which no any other messages were sent from FE to CE. The time interval is called FE heartbeat interval, and is set as below.

       Others - Reserved.

   . CE heartbeat dead interval - to tell FE the time interval the CE uses to check the CE liveness. If FE has not received any messages from CE for this time interval, FE may make a conclusion that the CE is dead or the association to the CE is lost. If set to 0, it means to choose a default value. (Comments: note that CE does not need to tell FE the CE heartbeat interval then.)

   . FE heartbeat interval - to command FE to use the interval for the FE Heartbeat Message generation. If set to 0, it means FE must choose a default value. (Comments: Note that CE does not need to set heartbeat dead interval for FE.)

</change to>

Based on above solution profile, I think the FE capability for heartbeat generation is still necessary. More explaination as:

   *  FE capability to generate Heatbeats: Incapable (value=0)or Capable(value=1). This capability actually tells CE if the FE is capable of actively generating Heartbeat Messages at a time intervals, which matches the FE heartbeat policy case 1. If incapable, the FE will not be able to support this policy.      

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Tue, 20 Sep 2005 12:31:15 -0400

Weiming,

On Tue, 2005-20-09 at 10:45 +0800, Weiming Wang wrote:
> Jamal,
> 
> I'm sorry to reply a bit late. I'm actually trying to understand 
> your meaning of FE MUST and CE MUST for heartbeat generating in 
> your response. I just found it is inconsistent in some way, therefore 
> hard to follow your meaning. I'm not sure if you actually mean "FE MUST 
> generate HBs" instead of "CE MUST". My general comment is, the scheme for 
> a "no heartbeat" of either CE or FE should be allowed, so as to support a 
> very confident system, where implementors believe they don't need the HBs 
> at all for the liveness check.
> 
> I see the main difference lying between us is, if you now agree to allow 
> the HB scheme as the FE automatically generates HB messages rather than 
> just resonse to CE HB messages? 
> 

I look at your text below and the only thing is i thought we said that
an FE MUST be able to generate HBs. i.e the capability is not needed.
I dont mind having it there, but i thought thats the decision we
reached, no?
The CE MAY configure the FE to use the HB. When it doesnt, it MAY
request for acknowledgement or never even send the HBs.
The CE MUST be able to generate HBs, it MAY not generate them if
configured so. 
We also said the "piggybacking" is a MUST. I think we need to describe
this mechanism once somewhere - give it a name maybe and stop repeating
its description everywhere (such as in the case you do in the CE/FE
heartbeat policy).

Other than that, i think we may need a FE Dead Interval as well to tell
the FE when the CE will declare the FE dead.

If you agree to this, please make changes to the HB section and send me
the XML. Then we can close this issue and section 3 (but not yet the HA
section until we fixup related things). 

cheers,
jamal

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 21 Sep 2005 11:24:32 +0800

----- Original Message ----- 
From: "Jamal Hadi Salim" <[email-masked]>
Subject: Re: Closure needed please comment WAS(Re: Section 9: Heartbeat Messages

> I dont mind having it there, but i thought thats the decision we
I also don't mind having it or not, i.e., we can implicitly assume that FE must have the capability to actively generate HB messages at intervals. My formal understanding was 'FE MUST support Heartbeat operations', that I thought meant FE must be able at least to respond HB request messages from CE.

> request for acknowledgement or never even send the HBs.
Agreed. I think the generic meaning has been included in the text with the FE heartbeat policy.

> The CE MUST be able to generate HBs, it MAY not generate them if
Agreed, and I think the meaning is also included.

> its description everywhere (such as in the case you do in the CE/FE
Sure, I try to add some text.

> the FE when the CE will declare the FE dead.
What the parameter can be used for in FE then, i.e., any usage for the parameter in FE?

> section until we fixup related things). 
Ok, and agreed.

Thanks,
Weiming

---

From: "Wang,Weiming" <[email-masked]>
Date: Wed, 21 Sep 2005 12:13:13 +0800

Hi all,

I have a general question on the FE protocol LFB and its attributes definitions in the protocol draft.

We have an XML in the Appendix. Can we understand that the attributes described in the draft Sections are just a  descriptive explaination on the XML definitions? Will the XML give the attribute value descriptions also?

I raise the question because I feel a little uncomfortable with current way in the draft to describe the attributes like the CE/FE heartbeat policy and CE/FE failover policy. 

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 21 Sep 2005 06:27:43 -0400

On Wed, 2005-21-09 at 11:24 +0800, Wang,Weiming wrote:
> I also don't mind having it or not, i.e., we can implicitly assume that FE 
> must have the capability to actively generate HB messages at intervals. 

This seemed to be the consensus at the concal.

> My 
> formal understanding was 'FE MUST support Heartbeat operations', that I 
> thought meant FE must be able at least to respond HB request messages from CE.
> 

Ok, my fault. Yes, formally if FE is able to respond or process HBs,
what you are saying is included.
So for simplicity lets still have 'FE MUST support Heartbeat operations'
to mean supporting generation and responding to HBs by the FE. This
means we are also reaching the consensus as per concal.

Send me the text privately when ready and i will release it to the list.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Wed, 21 Sep 2005 06:30:16 -0400

Weiming,

The XML is out-dated now that we have made changes to HB.
Can you also make changes to it when you change the HB text to reflect
more accurately reality?
Also if you dont mind looking at the draft and making sure text on FEPO
is consistent elsewhere on draft with these changes.

cheers,
jamal

On Wed, 2005-21-09 at 12:13 +0800, Wang,Weiming wrote:
> We have an XML in the Appendix. Can we understand that the attributes 
> described in the draft Sections are just a  descriptive explaination 
> on the XML definitions? Will the XML give the attribute value descriptions 
> also?
> 
> I raise the question because I feel a little uncomfortable with current way 
> in the draft to describe the attributes like the CE/FE heartbeat policy and 
> CE/FE failover policy. 
> 
> Thanks,
> Weiming

---

From: Weiming Wang <[email-masked]>
Date: Thu, 22 Sep 2005 01:03:23 +0800

----- Original Message -----
> Send me the text privately when ready and i will release it to the list.

Jamal, I think it may be better for you to keep consistency with your XML file
version maintainence by my sending you only the text instead of the XML. The
text regarding the Heartbeat attributes in FEPO is just as below:

Remove the following text in Section 6.2.1 on the FE Protocol Attributes
paragraph:

---
      *  Association Expiry Timer.  Defualt Value = 900 msec

      *  Heartbeat Interval.  Defualt Value = 300 msec
---
The reason to remove the Association Expiry Timer is, it can simply be replaced
by the 'CE heartbeat dead interval', I think. Any comments ?

Substitute it with the following text:

---

  . CE heartbeat policy - This policy, along with the parameter 'CE Heartbeat
Dead Interval (CE HDI)' as described below, actually tells the FE how to check
the CE liveness. The policy values with its meanings are listed as below:

         0(default) - This policy specifies that CE will send a Heartbeat
Message to FE whenever CE reaches a time interval within which no any other
messages were sent from CE to FE (Heartbeat generated in this way is sometimes
called Piggybacking-Heartbeat). A CE HDI parameter as described below is
associated with this policy. FE checks the CE being alive or dead just by
checking if it has or has not received any message (including Heartbeat Message
itself) for the CE HDI period. Note that CE can freely choose the time interval
to send Heartbeat messages and don't have to let FE know the interval, but
definitely, the time interval choosen must be smaller than the CE HDI assigned.

         1 - This policy specifies that CE will not send any Heartbeat Message
to FE in any case. This actually means CE does not want to ask FE to check the
CE liveness.

         Others - reserved.

   . FE heartbeat policy - This policy, along with the parameter 'FE Heartbeat
Interval (FE HI)', tells FE how it should act so that the CE it has associated
with can check the liveness of the FE based on the actions. The policy values
and the meanings are:

         0(default) - This policy actually specifies no extra action for FE on
its heartbeat generation. As a result, to let CE check its liveness, the FE may
only just act by responding to CE whenever CE sends Heartbeat Request Message
(also a normal HB message, only with whose ACK flag is set to AlwaysACK, see
Section 6.9) with a Heartbeat Response Message (also a normal HB message, only
with whose ACK flag is set to NoACK, see Section 6.9).

         1 - This policy specifies that FE must actively send a Heartbeat
Message if it reaches the time interval assigned by the FE HI followed, within
which no any other messages were sent from FE to CE.

         Others - Reserved.

   . CE Heartbeat Dead Interval (CE HDI) - to tell FE the time interval the FE
uses to check the CE liveness. If FE has not received any messages from CE for
this time interval, FE may make a conclusion that the CE is dead or the
association to the CE is lost.

   . FE Heartbeat Interval (FE HI) - to command FE to use the interval for the
FE Heartbeat Message generation. If set to 0, it means FE must choose a default
value. (Comments: Note that CE does not need to set heartbeat dead interval for
FE.)

---

Because of the change with the ACK flag usage in Heartbeat Message, Section 6.9
Heartbeat Message should also be changed. I propose the text as below:

---
6.9  Heartbeat Message

   The Heartbeat (HB) Message is used for one ForCES element (FE or CE)
   to asynchronously notify one or more other ForCES elements in the
   same ForCES NE on its liveness.

   A Heartbeat Message may be sent by CE or FE. The policies and the related
parameters for CE or FE to generate and send Heartbeat Messages are defined as
an attribute of FE protocol LFB, as described in Section 6.2.1 and also formally
defined in Appendix C.
   A Heartbeat Message is, in its PDU, only composed of a common header, with
the message body left empty. Detailed description of the message is as below.

   Message Transfer Direction:
       FE to CE, or CE to FE

   Message Header:
       The Message Type in the message header is set to MessageType =
'Heartbeat'.  The ACK flag in the header can be set to the value of 'NoACK' or
'AlwaysACK' as defined in Section 5.1. The default behavior for the ACK flag
that is set to any other value is defined just as the behavior of 'NoACK'. When
set to 'NoACK', the message will not expect any response from the receiver. When
set to 'AlwaysACK', the Heartbeat Message is then called a Heartbeat Request
Message, and will then expect a response from its receiver. There is no a
specific Heartbeat Response Message defined. A normal Heartbeat Message will act
as a response to a Heartbeat Request Message by its correlator set to that of
the responding Heartbeat Request Message and by its ACK flag always set to
'NoACK'. Current heartbeat policies as defined in Section 6.2.1 only allow a
Heartbeat Request Message being sent from CE to FE rather than contrary.

    Message Body:
       The message body is empty for the Heartbeat Message.

---

Thanks,
Weiming