Subject: [Megaco] Support for redundant MG failover

From: Flemming Andreasen <[email-masked]>
Date: Mon, 21 Apr 2003 10:59:27 -0400

Megaco supports some notion of redundant MGs and redundant MG failover,
however there is very little normative language, let alone explanation
of how this is supposed to work (and the current incarnation of the
upcoming Annex F doesn't help much here). The only model described is
the one in Section 11.4, where both the primary and secondary MG are
fully visible at all times and the MGC has to send duplicate messages to
the MG.  However, what about supporting models where the MGC does not
have to send multiple messages ? How does the protocol support the
following models (if at all):

1a) MG1 (primary) and MG2 (secondary) share the same IP-address with
complete state duplication (included for completeness since this one
will be 100% transparent to the MGC).
1b) MG1 (primary) and MG2 (secondary) share the same IP-address, and
there is complete Megaco state synchronization
1c) MG1 (primary) and MG2 (secondary) share the same IP-address, and
there is Megaco state synchronization at regular checkpoints (i.e. not
real-time so there may be some discrepances between the two)
2a) MG1 and MG2 have different IP addresses, and there is complete
Megaco state synchronization.
2b) MG1 and MG2 have different IP addresses, and there is Megaco state
synchronization at regular checkpoints (i.e. not real-time)

Thanks

        Flemming

_______________________________________________
Megaco mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/megaco

---

From: Christian Groves <[email-masked]>
Date: Tue, 22 Apr 2003 10:11:58 +1000

G'Day Fleming,

The megaco model can support the configurations that you list below. The
scenarios are implemented in such a way that the megaco interface is unaware
that state synchronisation is occuring. It's for this reason that the
specification doesn't say more about it, if you don't need Megaco messages then
there's no point putting it in a megaco specification.

Regards, Christian

---

From: Christian Groves <[email-masked]>
Date: Tue, 22 Apr 2003 10:11:58 +1000

G'Day Fleming,

Regards, Christian

---

From: Flemming Andreasen <[email-masked]>
Date: Mon, 21 Apr 2003 21:43:33 -0400

Hi Christian

> there's no point putting it in a megaco specification.

I agree with the last point, however it only applies to case 1a. Clearly something
has to happen in all the other cases, i.e. sending of a ServiceChange since the
failover is not 100% transparent and the MGC needs to be informed about the change
(whether it's a simple address change, mgcId change, or some degree of state loss).
As I noted before, Megaco is silent on this and if you were to conclude anything
from the current draft of Annex F, it would be that only model 1a is actually
supported today (at least in a standard manner). If you believe all of these models
actually are supported, then I'd like to know what ServiceChangeMethod,
ServiceChangeReason, and ServiceChangeMGCId (and/or possibly ServiceChangeAddress)
you would use in each case (apart from 1a that is).

-- Flemming

---

From: Flemming Andreasen <[email-masked]>
Date: Mon, 21 Apr 2003 21:43:33 -0400

Hi Christian

-- Flemming

---

From: Christian Groves <[email-masked]>
Date: Tue, 22 Apr 2003 15:08:43 +1000

G'Day Flemming,

May be my brain hasn't restarted after the Easter break

In cases other than 1a a failover method would be sent with the
ServiceChangeMGCId which the address of the association new or existing. How the
state synchronisation is done is still out of scope of Megaco. MG2 will either
have the up to date state knowledge (eg. duplicate megaco messages) or it will
have to audit or use some other means to find out.

Is it the state sharing part that you are questioning?

Regards, Christian

---

From: Christian Groves <[email-masked]>
Date: Tue, 22 Apr 2003 15:08:43 +1000

G'Day Flemming,

Regards, Christian

---

From: Flemming Andreasen <[email-masked]>
Date: Tue, 22 Apr 2003 09:50:50 -0400

Christian Groves wrote:

> ServiceChangeMGCId which the address of the association new or existing.

Something like that makes sense to me, however I don't find much support for it in the
H.248 specification. The text in H.248 (more specifically Section 11.4, all other
descriptions of "failover" applies to MGC related failures) suggests that failover is
something the failing MG sends, not the backup.  Again, taking a look at the upcoming
Annex F, I don't find any support for use of failover in these cases either.

> How the

Right, however it is imperative that the MGC has a way of figuring out whether state has
been lost or not, and hence can take appropriate recovery action, and that's the part
that seems to be somewhat missing.

> Is it the state sharing part that you are questioning?

That as well as the method, reason, and mgcID. Let's take the cases one by one and to
make things clearer let:
- domain(MG) be a single domain name for both MG1 and MG2
- domain(MG1) and domain(MG2) be a domain name for MG1 and MG2
- IP(MG1) and IP(MG2) be the IP address of MG1 and MG2

Also, in cases 1a to 1c, let's assume use of TCP. In cases 2a and 2b, let's assume use of
UDP.

Case 1a)     MG1 (primary) and MG2 (secondary) share the same IP-address with complete
state duplication (included for completeness since this one will be 100% transparent to
the MGC).

When MG1 fails, there will be no messages sent to the MGC, since there is complete state
duplication between MG1 and MG2 (incl. transport layer).

Case 1b)    MG1 (primary) and MG2 (secondary) share the same IP-address, and there is
complete Megaco state synchronization

MG1 crashes and hence cannot send any messages to the MGC. The MG1 TCP connection is
lost, and a new control association between MG2 and MGC must be created. MG2 sends a
ServiceChange with the following parameters:

mid:    domain(MG)
ServiceChangeMethod:    restart ?
ServiceChangeReason:    900 - service restored ?

Note that Annex F currently disallows failover/900, however restart/900 would be allowed.

Case 1c)    MG1 (primary) and MG2 (secondary) share the same IP-address, and there is
Megaco state synchronization at regular checkpoints (i.e. not real-time so there may be
some discrepancies between the two)

MG1 fails completetly and cannot send anything. MG2 sends a ServiceChange with the
following parameters:

mid:    domain(MG)
ServiceChangeMethod:    restart ?
ServiceChangeReason:     900/901/902/9xx ?

Again, failover cannot be combined with 900/901/902 according to current version of Annex
F.
900 would indicate no loss, i.e. latest checkpoint up to date.
901 would indicate total state loss, i.e. no checkpointing ever done
902 would indicate possible loss of transactions in process, but otherwise no state loss
9xx is a missing reason, that would indicate some level of state loss between  total
state loss (901) and only transaction in process state loss (902).

Case 2a)    MG1 and MG2 have different IP addresses, and there is complete Megaco state
synchronization.

MG1 fails completely and MG2 sends a ServiceChange with the following parameters:

mid:    domain(MG) ?
ServiceChangeMethod:    restart ?
ServiceChangeReason:    900 - service restored ?

This one is similar to 1b, but since we have different IP addresses there is another
issue. The mid in the message identifies this as the same MG, however we need to make
sure that messages are sent to MG1, not MG2 (recall we are using UDP now so can't depend
on the TCP connection implicitly handling this). How is this achieved (this problem
applies equally well to the initial registration by MG1) ?

Case 2b)    MG1 and MG2 have different IP addresses, and there is Megaco state
synchronization at regular checkpoints (i.e. not real-time)

mid:    domain(MG) ?
ServiceChangeMethod:    restart ?
ServiceChangeReason:    900/901/902/9xx ?

The issues here are the same as in 1c and 2a.

-- Flemming

---

From: Flemming Andreasen <[email-masked]>
Date: Tue, 22 Apr 2003 09:50:50 -0400

mid:    domain(MG)

mid:    domain(MG)

mid:    domain(MG) ?

mid:    domain(MG) ?

-- Flemming

---

From: Adrian Soncodi <[email-masked]>
Date: Tue, 22 Apr 2003 12:47:05 -0500

Hello, all,

I was wondering what is your oppinion on the following:

Another topic that is not addressed in the MEGACO spec is the case of a 
multi-homed MG, i.e. an MG where:
- the primary controller has 2 physical interfaces A1 and A2, 
potentially on 2 different subnets, working in load-sharing mode (rather 
than active-standby)
- the secondary MG controller can then be either transparrent (it takes 
over A1 and A2 on switchover) or non-transparent (has 2 more physical 
addresses, B1 and B2).

One may argue that multi-homing is not necessary, for example because 
you can always assign a single "logical" IP address A that is reachable 
over both interfaces A1 and A2. Yes, but this schema requires some 
dynamic IP routing capability in the backbone (e.g. to figure out when 
A1 is down and packets for A should be routed to A2), which has a number 
of disadvantages:
- Depending on the routing protocol, the latency for dynamic updates of 
the addresses can be very large (180 seconds in case of RIP), 
unacceptable for telephony.
- Depending on the network topology, interface failures may not become 
visible in the backbone. For example, if there is a LAN switch between 
the MG and the first router, then the router does not see when the 
interface A1 fails (unless the dynamic routing domain is extended to 
include the MG, which is not always the case).
- Some service providers (especially the smaller ones) do not run a 
dynamic IP routing protocol in their (whole) backbone, or they may not 
even have a real backbone, for example they may use leased lines, IP 
over ATM/FR PVCs with no sophisticated routing capabilities.

On the other hand, making the physical addresses A1 and A2 visible 
allows for redundant signaling paths to the MG using only static IP 
routing in the network, which in certain situations is desirable.

The MGC-MG redundancy schema gets even more complicated if you consider 
that the MGC can be multi-homed too. Effectively, it can have the same 
A1, A2, B1, B2 schema as the MG. In this case, what is really useful is 
an SCTP association (MGC-A1, MGC-A2, MG-A1, MG-A2), because this takes 
care of all the combinations. But if the MG supports only UDP, it is not 
clear from the standard how to emulate this SCTP behavior. For example:
- Some MGs do a DNS query for the MGC name, get 2 IP addresses in 
response, but they just ignore the second IP address, thus ignoring that 
the MGC is multi-homed.
- Other MGs consider the IP addresses as being 2 different MGCs, rather 
than one multi-homed MG. The problem is that they only talk to one MGC 
at a time, and then they need a "handover" to switch. So while talking 
on A1-A1, the MGC cannot for example audit on A2-A2, because this is 
immediately considered a failover (causing for example the pending 
notifications to be lost). The MGCP spec, for example, explicitly states 
that an MG should not change its controlling entity when being audited 
from another IP address (and it doesn't say whether this is a different 
MGC or the same multi-homed MGC).

I think what needs to be clarified here is:
- Whether an MG/MGC can be multi-homed and how to handle its multiple IP 
addresses.
- Whether an MG/MGC with active-standby controllers, transparent or 
non-transparent switchover, are considered one MG/MGC or 2 MG/MGCs
- Whether a domain name for an MG/MGC which yields more than one IP 
address through DNS, is associated with one controller, or with an 
active-standby pair, or possibly even with more active-standby pairs. 

The case with non-transparent MG controller switchover, as Flemming 
mentioned, poses additional problems. For example, if the active MG 
controller loses connectivity over A1 and A2, it may not even notice, 
because not all MGs do heartbeats. So if the MGC determines it can 
actually talk to B1 and B2, it cannot force an MG controller switchover, 
or at least the standard does not define the procedure for this. I would 
note that, in practice, however, most of the MGs and MGCs have 
transparent switchover (there is no B1, B2, they reuse A1, A2 ...).

The MGCP spec is a little more explicit on how the IP addresses are 
handled. So I don't understand why this topic is outside the scope of  
MEGACO.

I raised these issues because, from my experience, every single vendor 
interprets them differently, and MGC-MG redundancy is a delicate 
subject. Any suggestions?

Thanks

Adrian Soncodi,
Tekelec, Richardson, TX

---

From: Adrian Soncodi <[email-masked]>
Date: Tue, 22 Apr 2003 12:47:05 -0500

Hello, all,

Thanks

Adrian Soncodi,
Tekelec, Richardson, TX

---

From: Fernando Vallejo Lazaro <[email-masked]>
Date: Wed, 23 Apr 2003 10:40:43 +0200

Hi all,

Only some comments.

- You say : "So while talking on A1-A1, the MGC cannot for example audit
on A2-A2, because this is immediately considered a failover (...)". I
don't think this is true.
If an MG receives an Audit from MGC-A2 (which it does not know) it will
reject the transaction because there is no association established with
it.

- If the MG is multi-homed you can provision the two IP static addresses
as primary and secondary MGCs. In case one fails it will change to the
other IP which in fact is the
same MGC. Maybe there should be a deep explanation of how to do this to
smooth the process because the MG does not know whether the secondary is
an independent
MGC, a backup or a mirror and depending on this information it can
behave in one way or another (e.g. a ServiceChange Request only
Informational)

- Another thing the MGC can do when it detects the that A1 is going to
die is to send a Handoff to the interface A2 for each MG so that the
connections are restablished
in the new path.

Regards
Fernando

Adrian Soncodi wrote:

> Another topic that is not addressed in the MEGACO spec is the case of
> a multi-homed MG, i.e. an MG where:
> potentially on 2 different subnets, working in load-sharing mode
> (rather than active-standby)
> - the secondary MG controller can then be either transparrent (it
> takes over A1 and A2 on switchover) or non-transparent (has 2 more
> physical addresses, B1 and B2).
>
> you can always assign a single "logical" IP address A that is
> reachable over both interfaces A1 and A2. Yes, but this schema
> requires some dynamic IP routing capability in the backbone (e.g. to
> figure out when A1 is down and packets for A should be routed to A2),
> which has a number of disadvantages:
> - Depending on the routing protocol, the latency for dynamic updates
> of the addresses can be very large (180 seconds in case of RIP),
> The MGC-MG redundancy schema gets even more complicated if you
> consider that the MGC can be multi-homed too. Effectively, it can have
> the same A1, A2, B1, B2 schema as the MG. In this case, what is really
> useful is an SCTP association (MGC-A1, MGC-A2, MG-A1, MG-A2), because
> this takes care of all the combinations. But if the MG supports only
> UDP, it is not clear from the standard how to emulate this SCTP
> behavior. For example:
> response, but they just ignore the second IP address, thus ignoring
> that the MGC is multi-homed.
> - Other MGs consider the IP addresses as being 2 different MGCs,
> rather than one multi-homed MG. The problem is that they only talk to
> one MGC at a time, and then they need a "handover" to switch. So while
> talking on A1-A1, the MGC cannot for example audit on A2-A2, because
> this is immediately considered a failover (causing for example the
> pending notifications to be lost). The MGCP spec, for example,
> explicitly states that an MG should not change its controlling entity
> when being audited from another IP address (and it doesn't say whether
> this is a different MGC or the same multi-homed MGC).
>
> - Whether an MG/MGC can be multi-homed and how to handle its multiple
> IP addresses.
> actually talk to B1 and B2, it cannot force an MG controller
> switchover, or at least the standard does not define the procedure for
> this. I would note that, in practice, however, most of the MGs and
> MGCs have transparent switchover (there is no B1, B2, they reuse A1,
> A2 ...).
>
--
Fernando Vallejo Lazaro
Phone:  +34 91 330 5614
SW  Wireline ASD-Madrid
       ALCATEL

---

From: Fernando Vallejo Lazaro <[email-masked]>
Date: Wed, 23 Apr 2003 10:40:43 +0200

Hi all,

Only some comments.

Regards
Fernando

--
Fernando Vallejo Lazaro
Phone:  +34 91 330 5614
SW  Wireline ASD-Madrid
       ALCATEL

---

From: Adrian Soncodi <[email-masked]>
Date: Wed, 23 Apr 2003 15:45:38 -0500

Fernando,

Thanks for the reply. See comments inline.

Adrian

Fernando Vallejo Lazaro wrote:

> Hi all,
>
> Only some comments.
>
> - You say : "So while talking on A1-A1, the MGC cannot for example 
> audit on A2-A2, because this is immediately considered a failover 
> (...)". I don't think this is true.
> If an MG receives an Audit from MGC-A2 (which it does not know) it 
> will reject the transaction because there is no association 
> established with it.
>
Some MGs do this, others either don't reply or they failover. The latter 
may be in violation of the standard, but I think that's because it's not 
very well explained.

In my case, since I use the audits on A2-A2 as heartbeats (just to check 
for connectivity), any reply is OK, if only I get it. But this is a 
problem if some other MGC really wants to audit and is rejected. By the 
way, MGCP allows for this extra auditing, why is there a difference in 
approach? This only complicates things for an MGC which controls both 
MGCP and MEGACO MGs.

> - If the MG is multi-homed you can provision the two IP static 
> addresses as primary and secondary MGCs. In case one fails it will 
> change to the other IP which in fact is the
> same MGC.
>
Yes, but if the failure (MGCs interface A1) occurs after the initial 
ServiceChange, sometimes during operations when the MG does not have any 
notifications to send, and if the MG does not do any heartbeats, then it 
won't be aware of the failure, it will just think it's idle. So it will 
not initiate ServiceChange to the "alternate" MGC. Furthermore, even if 
the MG receives a new command from MGC A2, as you showed above, it will 
reply with an error and remain idle. (By the way, where in the standard 
is this rule stated?)

> Maybe there should be a deep explanation of how to do this to smooth 
> the process because the MG does not know whether the secondary is an 
> independent
Here, I concur with you and with Flemming, there should be more 
explanations. If you take into account multiple IP addresses, DNS names 
translating to one or more IP addresses, MGs and MGCs working in 
active-standby mode with or without complete state synchronization, as 
well as multiple active-standby pairs handoff, things get complicated 
enough to be worth additional explanations.

> connections are restablished
This is what it's forced to do, but the MG is not aware that the handoff 
is to the same MGC. Consequently, it continues to send pending 
notifications to the old IP address, as per RFC 3015 section 7.2.8 (and 
still in h248v2-04):

"At the same time, as indicated in clause 9, transaction replies and 
pending indications must be sent to the address from which the 
corresponding requests originated."

This forces the "new" MGC (which is not really new) to re-issue all the 
notification requests, which, apart from being a substantial overhead, 
can cause serious race conditions, especially with packages such as 
bcas. Again, from experience, this is a point of contention, whether or 
not it's the right behavior.

If the MG were somehow aware that this is the same MGC, the signaling 
path switchover between A1-A1 and A2-A2 would be faster and more reliable.

I think the MEGACO spec obviously did not consider multi-homed MGs/MGCs, 
therefore we are trying to see if the existing rules "happen to work" in 
this case. I have had a lot of arguments over it because of different 
interpretations.

> Fernando

Thanks again,

Adrian

>> - Depending on the network topology, interface failures may not 
>> become visible in the backbone. For example, if there is a LAN switch 
>> between the MG and the first router, then the router does not see 
>> when the interface A1 fails (unless the dynamic routing domain is 
>> extended to include the MG, which is not always the case).
>> dynamic IP routing protocol in their (whole) backbone, or they may 
>> not even have a real backbone, for example they may use leased lines, 
>> IP over ATM/FR PVCs with no sophisticated routing capabilities.
>>
>> consider that the MGC can be multi-homed too. Effectively, it can 
>> have the same A1, A2, B1, B2 schema as the MG. In this case, what is 
>> really useful is an SCTP association (MGC-A1, MGC-A2, MG-A1, MG-A2), 
>> because this takes care of all the combinations. But if the MG 
>> supports only UDP, it is not clear from the standard how to emulate 
>> this SCTP behavior. For example:
>> one MGC at a time, and then they need a "handover" to switch. So 
>> while talking on A1-A1, the MGC cannot for example audit on A2-A2, 
>> because this is immediately considered a failover (causing for 
>> example the pending notifications to be lost). The MGCP spec, for 
>> example, explicitly states that an MG should not change its 
>> controlling entity when being audited from another IP address (and it 
>> doesn't say whether this is a different MGC or the same multi-homed 
>> MGC).
>>
>> switchover, or at least the standard does not define the procedure 
>> for this. I would note that, in practice, however, most of the MGs 
>> and MGCs have transparent switchover (there is no B1, B2, they reuse 
>> A1, A2 ...).
>>
>> handled. So I don't understand why this topic is outside the scope 
>> of  MEGACO.
>>
>> I raised these issues because, from my experience, every single 
>> vendor interprets them differently, and MGC-MG redundancy is a 
>> delicate subject. Any suggestions?
>>
>> Thanks
>>
>> Adrian Soncodi,
>> Tekelec, Richardson, TX
>>  
>>  
>>  
>>
>> Christian Groves wrote:
>>
>>>G'Day Flemming,
>>>
>>>>>>1c) MG1 (primary) a
>>>>>>nd MG2 (secondary) 
>>>>>>share the same IP-address, and

---

From: Adrian Soncodi <[email-masked]>
Date: Wed, 23 Apr 2003 15:45:38 -0500

Fernando,

Adrian

Thanks again,

Adrian

---

From: Flemming Andreasen <[email-masked]>
Date: Thu, 24 Apr 2003 11:04:03 -0400

> active-standby pair, or possibly even with more active-standby pairs.

I agree - these are rather big holes in the current specs IMO.

> handled. So I don't understand why this topic is outside the scope of

Right. I have always thought it was unfortunate, that Megaco did not
specify anything about use of multiple IP addresses and/or interfaces
(apart from the static provisioning of backup MGCs) as this limits the
robustness of the protocol when using UDP/TCP. I was hoping that Annex F
would rectifiy some of that, but it doesn't seem like it.

> subject. Any suggestions?

1. Continued discussion on the mailing list (preferably with more input
from some of the traditional Megaco gurus).
2. Write a contribution for SG16 addressing this.
3. Forget about TCP and UDP and just use SCTP.

-- Flemming

---

From: Flemming Andreasen <[email-masked]>
Date: Thu, 24 Apr 2003 11:04:03 -0400

-- Flemming

---

From: Flemming Andreasen <[email-masked]>
Date: Thu, 24 Apr 2003 11:23:50 -0400

Adrian Soncodi wrote:

> Some MGs do this, others either don't reply or they failover. The
> latter may be in violation of the standard, but I think that's because
> it's not very well explained.
>

I think part of the problem here is that the concept of a control
association is not very well defined when it comes to UDP; how do you
determine if a message received is part of the UDP control association ?

>
> In my case, since I use the audits on A2-A2 as heartbeats (just to
> check for connectivity), any reply is OK, if only I get it. But this
> is a problem if some other MGC really wants to audit and is rejected.
> By the way, MGCP allows for this extra auditing, why is there a
> difference in approach? This only complicates things for an MGC which
> controls both MGCP and MEGACO MGs.
>
> ServiceChange, sometimes during operations when the MG does not have
> any notifications to send, and if the MG does not do any heartbeats,
> then it won't be aware of the failure, it will just think it's idle.
> So it will not initiate ServiceChange to the "alternate" MGC.
> Furthermore, even if the MG receives a new command from MGC A2, as you
> showed above, it will reply with an error and remain idle. (By the
> way, where in the standard is this rule stated?)

Yes, however there are ways around this problem (by design, Megaco chose
to only have the MG be able to initiate the control association, so you
just need to make sure it does that - see H.248.14 for a solution).

There are however a couple of bad things here:
1. Only the statically provisioned list of backup MGCs can be contacted,
and only in the order they were provisioned.
2. The spec is silent on whether a backup MGC replicates the state of
the primary or not. Consequently, you cannot assume that it does, and
hence switching to the backup is something you don't want to do
prematurely.
3. The switchover is not graceful, i.e. there has to be a failure before
you start contacting the backup. In contrast, a more robust redundancy
scheme would detect/suspect the failure earlier during retransmission
and switch to one of the backups to prevent any loss of state.
Furthermore, there should be a way of indicating that a backup is fully
redundant and hence the switchover can be done fairly early without any
risk.

> explanations. If you take into account multiple IP addresses, DNS
> names translating to one or more IP addresses, MGs and MGCs working in

Agreed - there is work to be done in this area.

>> - Another thing the MGC can do when it detects the that A1 is going
>> to die is to send a Handoff to the interface A2 for each MG so that
>> the connections are restablished
>> in the new path.
>
> This is what it's forced to do, but the MG is not aware that the
> handoff is to the same MGC. Consequently, it continues to send pending
> notifications to the old IP address, as per RFC 3015 section 7.2.8
> (and still in h248v2-04):
>
> This forces the "new" MGC (which is not really new) to re-issue all
> the notification requests, which, apart from being a substantial
> overhead, can cause serious race conditions, especially with packages
> such as bcas. Again, from experience, this is a point of contention,
> whether or not it's the right behavior.
>
> I think the MEGACO spec obviously did not consider multi-homed
> MGs/MGCs, therefore we are trying to see if the existing rules "happen
> to work" in this case. I have had a lot of arguments over it because
> of different interpretations.
>

I agree (and actually, I'm surprised nobody has addressed it yet because
it has been known for a long time and pops up on the mailing list from
time time, but without much progress being made).

-- Flemming

---

From: Flemming Andreasen <[email-masked]>
Date: Thu, 24 Apr 2003 11:23:50 -0400

-- Flemming