Subject: [Megaco] Support for redundant MG failover

From: Flemming Andreasen <[email-masked]>
Date: Mon, 21 Apr 2003 10:59:27 -0400

Megaco supports some notion of redundant MGs and redundant MG failover,
however there is very little normative language, let alone explanation
of how this is supposed to work (and the current incarnation of the
upcoming Annex F doesn't help much here). The only model described is
the one in Section 11.4, where both the primary and secondary MG are
fully visible at all times and the MGC has to send duplicate messages to
the MG.  However, what about supporting models where the MGC does not
have to send multiple messages ? How does the protocol support the
following models (if at all):

1a) MG1 (primary) and MG2 (secondary) share the same IP-address with
complete state duplication (included for completeness since this one
will be 100% transparent to the MGC).
1b) MG1 (primary) and MG2 (secondary) share the same IP-address, and
there is complete Megaco state synchronization
1c) MG1 (primary) and MG2 (secondary) share the same IP-address, and
there is Megaco state synchronization at regular checkpoints (i.e. not
real-time so there may be some discrepances between the two)
2a) MG1 and MG2 have different IP addresses, and there is complete
Megaco state synchronization.
2b) MG1 and MG2 have different IP addresses, and there is Megaco state
synchronization at regular checkpoints (i.e. not real-time)

Thanks

        Flemming

_______________________________________________
Megaco mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/megaco

---

From: "Kevin Boyle" <[email-masked]>
Date: Thu, 24 Apr 2003 12:32:52 -0400

Comments inline. [KJBII]

Kevin

-----Original Message-----
From: Flemming Andreasen [mailto:[email-masked]] 
Sent: Thursday, April 24, 2003 11:24 AM
To: Adrian Soncodi
Cc: Fernando Vallejo Lazaro; [email-masked]
Subject: Re: [Megaco] Support for redundant MG failover

Adrian Soncodi wrote:

> Fernando,
>
> Thanks for the reply. See comments inline.
>
> Adrian
>
>
>
> Fernando Vallejo Lazaro wrote:
>
>> Hi all,
>>
>> Only some comments.
>>
>> - You say : "So while talking on A1-A1, the MGC cannot for example 
>> audit on A2-A2, because this is immediately considered a failover 
>> (...)". I don't think this is true. If an MG receives an Audit from 
>> MGC-A2 (which it does not know) it will reject the transaction 
>> because there is no association established with it.
>
> Some MGs do this, others either don't reply or they failover. The 
> latter may be in violation of the standard, but I think that's because 
> it's not very well explained.
>

I think part of the problem here is that the concept of a control
association is not very well defined when it comes to UDP; how do you
determine if a message received is part of the UDP control association ?

[KJBII] H.248 control associations have absolutely nothing to do with
transport.  To quote from the most recent version of Annex F: "A control
association is a communication relationship whereby an MGC is controlling an
MG.  A control association is instantiated via registration.  The control
association is terminated by the MG going out of service, being successfully
handed off to an alternate MGC, or successfully failing over to an alternate
MGC.  An MG shall have at most one control association at any time.  Only
MGs may instantiate control associations.  Control associations as defined
here apply only within the context of media gateway control signalling.
This definition should not be confused with that of control associations in
the context of transport (TCP, UDP, etc.)."

[KJBII] So, to go even further on the point above, the MG may "know" MGC-A2
(that is, have it in its list of MGCs), but because there can be only one
control association at any given time, and those associations are
established by the MG, the communication from MGC-A2 should be viewed as, at
best, an inappropriate control message.  At worst, it would be an attack to
attempt to gain control of the MG.  Also, I would like to note that an audit
command would never be construed as a failover.  Only a SC with a method of
failover would be a failover.

>
> In my case, since I use the audits on A2-A2 as heartbeats (just to 
> check for connectivity), any reply is OK, if only I get it. But this 
> is a problem if some other MGC really wants to audit and is rejected. 
> By the way, MGCP allows for this extra auditing, why is there a 
> difference in approach? This only complicates things for an MGC which 
> controls both MGCP and MEGACO MGs.

[KJBII] Megaco and MGCP are not the same.  There are many differences
between the two protocols, and this just happens to be one of them.

>
>> - If the MG is multi-homed you can provision the two IP static 
>> addresses as primary and secondary MGCs. In case one fails it will 
>> change to the other IP which in fact is the same MGC.
>
> Yes, but if the failure (MGCs interface A1) occurs after the initial 
> ServiceChange, sometimes during operations when the MG does not have 
> any notifications to send, and if the MG does not do any heartbeats, 
> then it won't be aware of the failure, it will just think it's idle. 
> So it will not initiate ServiceChange to the "alternate" MGC. 
> Furthermore, even if the MG receives a new command from MGC A2, as you 
> showed above, it will reply with an error and remain idle. (By the 
> way, where in the standard is this rule stated?)

Yes, however there are ways around this problem (by design, Megaco chose to
only have the MG be able to initiate the control association, so you just
need to make sure it does that - see H.248.14 for a solution).

There are however a couple of bad things here:
1. Only the statically provisioned list of backup MGCs can be contacted, and
only in the order they were provisioned. 

[KJBII] This is bad because?

2. The spec is silent on whether a backup MGC replicates the state of the
primary or not. Consequently, you cannot assume that it does, and hence
switching to the backup is something you don't want to do prematurely. 

[KJBII] Why is this the MGs concern?  The backup MGC is responsible for
getting itself attuned to the current state of affairs on the MG.  This can
be done in any number of ways, some of which are more detrimental to
existing connections than others.

3. The switchover is not graceful, i.e. there has to be a failure before you
start contacting the backup. In contrast, a more robust redundancy scheme
would detect/suspect the failure earlier during retransmission and switch to
one of the backups to prevent any loss of state. Furthermore, there should
be a way of indicating that a backup is fully redundant and hence the
switchover can be done fairly early without any risk.

[KJBII] Hmmm.  How would the MG know that the retransmissions were due to
MGC failure, as opposed to network congestion?  I believe that the MG will
have to wait for the retransmissions to run their course, or to utilize a
different method of determining that the MGC had failed that eliminates
network congestion as a possibility.

>> Maybe there should be a deep explanation of how to do this to smooth 
>> the process because the MG does not know whether the secondary is an 
>> independent MGC, a backup or a mirror and depending on this 
>> information it can behave in one way or another (e.g. a ServiceChange 
>> Request only
>> Informational)
>
> Here, I concur with you and with Flemming, there should be more 
> explanations. If you take into account multiple IP addresses, DNS 
> names translating to one or more IP addresses, MGs and MGCs working in 
> active-standby mode with or without complete state synchronization, as 
> well as multiple active-standby pairs handoff, things get complicated 
> enough to be worth additional explanations.

Agreed - there is work to be done in this area.

>> - Another thing the MGC can do when it detects the that A1 is going 
>> to die is to send a Handoff to the interface A2 for each MG so that 
>> the connections are restablished in the new path.
>
> This is what it's forced to do, but the MG is not aware that the 
> handoff is to the same MGC. Consequently, it continues to send pending 
> notifications to the old IP address, as per RFC 3015 section 7.2.8 
> (and still in h248v2-04):
>
> "At the same time, as indicated in clause 9, transaction replies and 
> pending indications must be sent to the address from which the 
> corresponding requests originated."
>
> This forces the "new" MGC (which is not really new) to re-issue all 
> the notification requests, which, apart from being a substantial 
> overhead, can cause serious race conditions, especially with packages 
> such as bcas. Again, from experience, this is a point of contention, 
> whether or not it's the right behavior.

[KJBII] Not true.  Any NEW event detections would go to the "new" MGC.
Remember, events in H.248 persist until a new events descriptor is sent to
replace the old.  Therefore, even through a handoff/failover, the MG should
continue to scan for events and report them to the controlling MGC -- the
"new" MGC once the handoff/failover occurs.

>
> If the MG were somehow aware that this is the same MGC, the signaling 
> path switchover between A1-A1 and A2-A2 would be faster and more 
> reliable.
>
> I think the MEGACO spec obviously did not consider multi-homed 
> MGs/MGCs, therefore we are trying to see if the existing rules "happen 
> to work" in this case. I have had a lot of arguments over it because 
> of different interpretations.
>

I agree (and actually, I'm surprised nobody has addressed it yet because it
has been known for a long time and pops up on the mailing list from time
time, but without much progress being made).

-- Flemming

[KJBII] There are many possibilities for redundancy -- multihoming is not
the only one.  Further, I don't see how the rules don't work, or how we can
optimize this particular case without penalizing the other possibilities.
The MG should not know, or care, that the MGC is multihomed.  The same goes
for the MGC with respect to the MG.  If you change IP addresses, then you
change control association, and that has consequences that have to be
rectified by the MGC.