Subject: HA and Heartbeats

From: Jamal Hadi Salim <[email-masked]>
Date: Fri, 16 Sep 2005 16:08:56 -0400

While correlating different sections:

--> If a CE and an FE are each sending heartbeats to the other, are they
required to use the same values for HB Interval and HB Dead Interval?
If not we need to be creative with FEPO. I know it's slightly out of
date at the moment, just trying to capture issues.

- If an FE is connected to multiple CEs in a CElist for HA purposes and
it is associated to all of them (clearly amongst the CEs, theres only
one which is a master at a time), on heartbeats:
        -> Do we allow the FE send heartbeats to the non-primary CEs?
        -> Do we allow the non-primary CEs send heartbeats to the FE?
        -> If so, are the CEs required to use the same values for HBI
and HBDI?

I have some thoughts - but wanted to solicit input first.

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 16 Sep 2005 21:08:02 -0400

It seems that no matter what, the CE has to configure the FE heartbeat 
interval.
The CE has to also configure the FE as to its timeout for declaring the CE 
dead.
The CE also must have local configuration as to what its heartbeat transmit 
interval and its timeout for declaring the FE dead are.
I am inclined not to couple these two.

However, it does not seem to change the protocol or the FE Protocol Object 
anyway.

Yours,
Joel

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sat, 17 Sep 2005 07:30:08 -0400

The concern is on the case of multiple CEs associating to a single FE
(as we defined in section 8 for HA). As described in that section,
only one of those CEs is actually configuring the FE. Does the FE need
to know of the liveliness of all those CEs (that it associated with)? If
yes, then the simplest thing would be at least multiple FEPO LFB
_instances_, no? Upto this point we have been saying there is only one
instance of the FEPO per FE. 
Further, if we are saying that only the primary CE is the configurator
of the FE, then if we create instances of FEPO - shouldnt we have
multiple configurators of at least that one LFB?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sat, 17 Sep 2005 10:25:41 -0400

An interesting question that had not occurred to me.
We have tried up till now to keep the impact of HA to a minimum for this 
revision, to keep our lives simple.
In fact, I had assumed (possibly incorrectly) that heartbeat was only 
exchanged with the primary, active, CE.
Otherwise, we are going to start need explicit configuration information 
related to HA support.

Yours,
Joel M. Halpern

---

From: Jamal Hadi Salim <[email-masked]>
Date: Sun, 18 Sep 2005 09:25:19 -0400

> exchanged with the primary, active, CE.

We need to make a decision of whether this is what we want to do.

> Otherwise, we are going to start need explicit configuration information 

We could have all the CEs checking liveliness via heartbeats and that we
allow them only to configure/control an instance of FEPO and nothing
more.
Otherwise it reduces the value of having multiple CEs for HA purposes.

Infact, would it be unreasonable to expect multiple CEs as masters of
different (non-conflicting) LFB instances?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Sun, 18 Sep 2005 18:40:58 -0400

We have currently declared that this must be done as separate 
LFBs.  Otherwise you have the problem of which CE sets up the data path.
Yours,
Joel

At 09:25 AM 9/18/2005, Jamal Hadi Salim wrote:
>jama

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 19 Sep 2005 08:19:50 -0400

> LFBs.  Otherwise you have the problem of which CE sets up the data path.

Ok, true i remember that we reached such a consensus. Is it captured in
any text anywhere? 
On the heartbeats, the way i see it is we have two options:

a) no heartbeats between FE-CE. Clearly this means no knowledge of the
availability of the backup CE(s) PL layer. It would mean more needs to
be done at the FE in terms of signaling on failover.

b) We allow the CEs to only configure their FEPO instance. This violates
slightly the decision reached so far. Only difference is that the backup
CEs are not configuring the datapath.

Any other alternatives?

cheers,
jamal

---

From: "Joel M. Halpern" <[email-masked]>
Date: Mon, 19 Sep 2005 10:07:23 -0400

I am not sure it is any better, but I can think of one other alternative.
For compelteness...
The configuration of information about relations with backup CEs could be 
the domain of the FEM, and therefore out of scope for the work for 
now.  This would mean that backup CEs could not configure the heartbeat / 
livness interval.

Yours,
Joel

---

From: Weiming Wang <[email-masked]>
Date: Mon, 19 Sep 2005 22:57:21 +0800

----- Original Message ----- 
From: "Jamal Hadi Salim" <[email-masked]>
Subject: HA and Heartbeats

> date at the moment, just trying to capture issues.
My thought is, HB Interval and dead interval can be different. They can be differently assigned to FE by CE via the CE/FE heartbeat policy attribute easily. 

>         -> Do we allow the FE send heartbeats to the non-primary CEs?
No, I think. A non-primary CE does not need to FE aliveness when it is still stand-by.

>         -> Do we allow the non-primary CEs send heartbeats to the FE?
For the simplicity consideration, I think we'd better not set so. The only function of non-primary CEs sending heartbeats to FE is to let the FE know the non- primary CE aliveness so that the FE might use the info in its HA policy. But, I really don't think we need this in current HA scheme. 

>         -> If so, are the CEs required to use the same values for HBI
Then, we don't have such care.

Thanks,
Weiming

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 19 Sep 2005 11:27:00 -0400

On Mon, 2005-19-09 at 10:07 -0400, Joel M. Halpern wrote:
[..]
> liveness interval.

Ok, could i get two more people to agree to this?

cheers,
jamal

---

From: Weiming Wang <[email-masked]>
Date: Mon, 19 Sep 2005 23:39:07 +0800

----- Original Message ----- 
From: "Weiming Wang" <[email-masked]>
> My thought is, HB Interval and dead interval can be different. They can be differently assigned to FE by CE via the CE/FE heartbeat policy attribute easily. 

A correction on this, I mean the HB Interval/dead interval for CE and for FE can be different.

Thanks,
Weiming

---

From: Avri Doria <[email-masked]>
Date: Mon, 19 Sep 2005 18:08:05 +0200

I agree with the idea that FE should only configure and send HB to  
the controlling CE.  And that the backup CE should not be configuring/ 
sending HB to the FEs.

It is up to the CEs to have coordinated information.
The CE  should have a means of communicating the state of the FE  
under their care.  Of course this work is not currently in scope.

a.

---

From: Robert Haas <[email-masked]>
Date: Mon, 19 Sep 2005 18:13:59 +0200

This would mean that when changing from the primary to the backup CE, 
the FE would have to first setup an association with the backup CE. The 
reason I say this is because I don't think an association without 
liveliness checks (heartbeat excahnges) makes much sense.

Regards,
-Robert

-- 
Dr. Robert R. Haas
IBM Zurich Research Laboratory
Säumerstrasse 4
CH-8803 Rüschlikon/Switzerland
phone +41-44-724-8698  fax +41-44-724-8578  http://www.zurich.ibm.com/~rha

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 19 Sep 2005 12:12:37 -0400

> under their care.  Of course this work is not currently in scope.

Thanks Avri - I think we have a hum; i will capture this.

cheers,
jamal

---

From: Jamal Hadi Salim <[email-masked]>
Date: Mon, 19 Sep 2005 12:19:02 -0400

> liveliness checks (heartbeat excahnges) makes much sense.

I think what you are saying also makes sense - even though unfortunately 
we are in a rush to close this issue.
Could other people comment on this?

Note, if one was to use a CEID that is multicast, then all these issues
will disappear i.e some repsonsibilities shift to the TML; the FE is
oblivious about who sends or receives HBs or which real CE was
associated to. Of course this model is not disallowed by current setup.

cheers,
jamal

---

From: Avri Doria <[email-masked]>
Date: Tue, 20 Sep 2005 14:44:25 +0200

Hi,

I don't see why it doesn't make sense.  It seems to me that the  
association could be setup and be in a standby mode.  This means that  
when operating in an HA mode and contact  is lost, then the CE that  
becomes the new primary could immediately send a HB with request for  
ACK.  And likewise when an FE lost HB with its primary it would know  
who to send a HB to, using a default HB mechanism.

without preset associations, how would know where to make reparative  
contact without going back to the still nonexistent CEM/FEM?

a.

On 19 sep 2005, at 18.13, Robert Haas wrote:

> This would mean that when changing from the primary to the backup  
> CE, the FE would have to first setup an association with the backup  
> CE. The reason I say this is because I don't think an association  
> without liveliness checks (heartbeat excahnges) makes much sense.
>
> Regards,
> -Robert
>
> Jamal Hadi Salim wrote:
>
>
>>> The configuration of information about relations with backup CEs  
>>> could be the domain of the FEM, and therefore out of scope for  
>>> the work for now.  This would mean that backup CEs could not  
>>> configure the heartbeat / liveness interval.
>>>
>>>
>>>
>>

---

From: Robert Haas <[email-masked]>
Date: Tue, 20 Sep 2005 15:47:04 +0200

Hi Avri,
What if in addition to sending HBs for the first time to its backup CE, 
the FE would have to setup the association first ? Then all that is 
needed is the CE ID of the backup CE to be stored in the FEPO.
I am not comfortable with a "dormant" type of association, i.e., an 
association that has been setup (together with a working TML), but for 
which no PL HB is flowing; the TML will be working anyway. So I'd prefer 
that either we keep the association alive with PL HBs too, or we don't 
set it at all before the switch to the backup CE occurs.

Regards,
-Robert

Avri Doria wrote:

>>>> could be the domain of the FEM, and therefore out of scope for  the 
>>>> work for now.  This would mean that backup CEs could not  configure 
>>>> the heartbeat / liveness interval.
>>>>
>>>>
>>>>
>>>