Subject: [dhcwg] dhcp-failover: the system won't be recovered. How to address it?

From: Toyo Abe <[email-masked]>
Date: Thu, 03 Dec 2009 18:04:33 +0900

Greetings,

I have a question about DHCP-Failover protocol, which is an old
draft (draft-ietf-dhc-failover-12.txt) in the IETF.

I uses ISC DHCP, which is a reference implementation of DHCP,
because it supports DHCP-FO protocol too.
I posted the same report to ISC DHCP mailing list, and then
I was suggested to repost it to the dhc wg.

I've encountered problems in DHCP-FO feature of ISC DHCP.
I wonder if you could help me with resolving the problem. 

There are two cases that the system won't be recovered.
(ISC DHCP version is cvs HEAD) 

*SCENARIO 1*

The following message was shown and both of the primary and the secondary
looked like out-of-service. And they never got recovered. 
---------------------
DHCPDISCOVER from 00:0c:29:e9:39:0c via eth1: not responding (resolving conflicts) 
---------------------

   Primary                                  Secondary

      |                                         |
POTENTIAL-CONFLICT                        POTENTIAL-CONFLICT
      |                                         |
      | >-- UPDREQ ----------------->           |
      |                                         |
      |         <------------------- BNDUPD --< |
      | >-- BNDACK ----------------->           |
     ...                                       ...
      |                                         |
      |         <------------------- BNDUPD --< |
      | <<========= Connection lost =========>> |
      |                                         |
     ...                                       ...
      |                                         |
RESOLUTION-INTERRUPTED                    RESOLUTION-INTERRUPTED
      |                                         |
      | >-- CONNECT ---------------->           |
      |         <--------------- CONNECTACK --< |
      |                                         |
      | <<===== Connection established ======>> |
      |                                         |
POTENTIAL-CONFLICT                        POTENTIAL-CONFLICT
      |                                         |
      | >-- (Doesn't send UPDREQ) --> X         |
      |                                         |
      | >-- CONTACT ---------------->           |
      |         <------------------ CONTACT --< |
      |                                         |
      | >-- CONTACT ---------------->           |
      |         <------------------ CONTACT --< |
      |                                         |
      | >-- CONTACT ---------------->           |
      |         <------------------ CONTACT --< |
      |                                         |
     ...                                       ...

The primary seems to continue waiting for BNDUPD or UPDDONE from the secondary.
OTOH, the secondary continues waiting for UPDREQ from the primary. 
The both servers, eventually, remain POTENTIAL-CONFLICT forever.

In the above case, ISC DHCP doesn't allow the server to resend UPDREQ and
expect the rest of the update process to be proceeded.
Although this might be an implementation specific problem, the spec of DHCP-FO
doesn't mention the above case so I want to clarify how this should be addressed.

*SCENARIO 2* 

This one also is the case that the system won't be recovered.

Primary                                  Secondary

   |                                         |
POTENTIAL-CONFLICT                        POTENTIAL-CONFLICT
   |                                         |
   | >-- UPDREQ ----------------->           |
   |                                         |
   |         <------------------- BNDUPD --< |
   | >-- BNDACK ----------------->           |
  ...                                       ...
   |                                         |
   |         <------------------ UPDDONE --< |
   |                                         |
CONFLICT-DONE                                |
   |                                         |
   |         <------------------- UPDREQ --< |
   |                                         |
   | >-- BNDUPD ----------------->           |
   |         <------------------- BNDACK --< |
  ...                                       ...
   |                                         |
   | <<========= Connection lost =========>> |
   |                                         |
  ...                                       ...
   |                                         |
   |                                   RESOLUTION-INTERRUPTED
   |                                         |
   | >-- CONNECT ---------------->           |
   |         <--------------- CONNECTACK --< |
   |                                         |
   | <<===== Connection established ======>> |
   |                                         |
   |                                   POTENTIAL-CONFLICT
   |                                         |
   |         X <-- (Doesn't send UPDREQ) --< |
   |                                         |
   | >-- CONTACT ---------------->           |
   |         <------------------ CONTACT --< |
   |                                         |
   | >-- CONTACT ---------------->           |
   |         <------------------ CONTACT --< |
   |                                         |
   | >-- CONTACT ---------------->           |
   |         <------------------ CONTACT --< |
   |                                         |
  ...                                       ...

Furthermore, I saw the following error msg on the primary
when I tested the case above.
---------------------
Peer adapter: Invalid attempt to move from potential-conflict to resolution-interrupted while local state is conflict-done. 
---------------------
ISC DHCP treats it's invalid that the secondary moves from potential-conflict
to resolution-interrupted while the primary is in conflict-done.
Once this happens, the secondary would not be reintegrated to the system.

I want to clarify how the servers should work in this case.
The spec says just 'remain in CONFLICT-DONE state until the condition for
transition out become satisfied'. 
The condition is that the secondary has made a transition into NORMAL state.
However, as you can see, the secondary never move to NORMAL in this case.

I made some changes locally to ISC DHCP, which is attached the email for
reference.
The changes are the followings.
 - allow to resend UPDREQ when the cases I reported here happen
 - allow the secondary to move its state from potential-conflict to
   resolution-interrupted while the primary is in conflict-done.

The attached works fine in-house but I really want to clarify how these cases
should be resolved w/o breaking the protocol.
Could someone give me advice about how this problem should be solved? 

Sincerely,
 -toyo

---
 server/failover.c |   54 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 53 insertions(+), 1 deletion(-)

--- DEST.orig/server/failover.c	2009-10-02 18:31:01.000000000 +0900
+++ DEST/server/failover.c	2009-10-05 13:52:29.000000000 +0900
@@ -1492,6 +1492,34 @@ isc_result_t dhcp_failover_state_signal 
 	return ISC_R_SUCCESS;
 }
 
+isc_result_t dhcp_failover_purge_ack_queue (dhcp_failover_state_t *state)
+{
+	struct lease *l, *n;
+
+	if (state->ack_queue_head) {
+		lease_reference (&l, state->ack_queue_head, MDL);
+		lease_dereference (&state->ack_queue_head, MDL);
+		do {
+			l->flags &= ~ON_ACK_QUEUE;
+			if (l->next_pending) {
+				lease_reference (&n,
+						 l->next_pending, MDL);
+				lease_dereference (&l->next_pending, MDL);
+			}
+			lease_dereference (&l, MDL);
+			if (n) {
+				lease_reference (&l, n, MDL);
+				lease_dereference (&n, MDL);
+			}
+		} while (l);
+		lease_dereference (&state->ack_queue_tail, MDL);
+	}
+
+	state->cur_unacked_updates = 0;
+
+	return ISC_R_SUCCESS;
+}
+
 isc_result_t dhcp_failover_state_transition (dhcp_failover_state_t *state,
 					     const char *name)
 {
@@ -1532,6 +1560,19 @@ isc_result_t dhcp_failover_state_transit
 			return ISC_R_SUCCESS;
 		
 		      case potential_conflict:
+			if (state->curUPD) {
+				/*
+				 * In case that disconnect event happens while resolving
+				 * bind information with a partner, clear curUPD so that
+				 * the process can be restarted from the beginning.
+				 */
+				state->curUPD = 0;
+				/*
+				 * And purge ack_queue since we're not sure if they
+				 * are on the partner's toack_queue.
+				 */
+				dhcp_failover_purge_ack_queue(state);
+			}
 			return dhcp_failover_set_state
 				(state, resolution_interrupted);
 				
@@ -1852,7 +1893,9 @@ isc_result_t dhcp_failover_set_state (dh
 	    break;
 
 	  case potential_conflict:
-	    if (state -> i_am == primary)
+	    if (state->i_am == primary ||
+		(state->i_am == secondary &&
+		 state->partner.state == conflict_done))
 		    dhcp_failover_send_update_request (state);
 	    break;
 
@@ -2171,6 +2214,15 @@ isc_result_t dhcp_failover_peer_state_ch
 			dhcp_failover_set_state(state, new_state);
 			break;
 
+		      case potential_conflict:
+		      case resolution_interrupted:
+			/*
+			 * This can happen when the connection lost and recovered
+			 * while only the secondary in potential-conflict.
+			 * In that case, we have to remain in conflict-done.
+			 */
+			break;
+
 		      default:
 			log_fatal("Peer %s: Invalid attempt to move from %s "
 				"to %s while local state is conflict-done.",

---

From: Johan Bosaeus <[email-masked]>
Date: Mon, 7 Dec 2009 16:53:58 +0100

On Thursday 03 December 2009 10:04:33 am Toyo Abe wrote:
> DHCPDISCOVER from 00:0c:29:e9:39:0c via eth1: not responding (resolving
> conflicts) ---------------------
>
>
>    Primary                                  Secondary
>
>
>       |
>       |
>       |
>       |
>       | >
>       |
>       |
>       |
>       |
>       |
We have had the same problems with the spec. Indeed, I would like to widen the 
problem. You cannot be sure that the UPDREQ command actually reaches the 
SECONDARY and gets processed there although the TCP connection is up and 
running as well as the "heartbeat" - CONTACT messages interchange -.  

In this case, we simply resend the UPDREQ command each time we are transiting 
to the POTENTIAL_CONFLICT state.  The receiving SECONDARY in this case is 
resilient to multiple UPDREQ:s. The important thing is that the PRIMARY 
finally obtains the UPDDONE command from the SECONDARY, so it is able to 
transit to next state, CONFLICT_DONE.

>      ...                                       ... 
>
> The primary seems to continue waiting for BNDUPD or UPDDONE from the
> secondary. OTOH, the secondary continues waiting for UPDREQ from the
> primary. The both servers, eventually, remain POTENTIAL-CONFLICT forever.
>
> Although this might be an implementation specific problem, the spec of
> DHCP-FO doesn't mention the above case so I want to clarify how this should
> be addressed.
>
>
>
> *SCENARIO 2*
>
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
>    |
   ...                                       ...
Same thing here.  You cannot assume  whether the previously sent UPDREQ has 
been received or not. Sure, the UPDREQ has been received it this case, but in 
a state, CONFLICT_DONE, which the implementation fails to remember. The draft 
doesn't specify it either, so I guess the intention of the draft is that both 
server's should start all over again in POTENTIAL_CONFLICT state.  If so, we 
may end up in an almost infinite loop when there are *lots* of bindings to 
recover and the connection is unstable.

At any rate, the draft standard you refer to was never accepted, so the issue 
might be better asked on the ISC mailing list.

> Peer adapter: Invalid attempt to move from potential-conflict to
> resolution-interrupted while local state is conflict-done.
> ---------------------
> ISC DHCP treats it's invalid that the secondary moves from
> potential-conflict to resolution-interrupted while the primary is in
> conflict-done.
> The condition is that the secondary has made a transition into NORMAL
> state. However, as you can see, the secondary never move to NORMAL in this
> case.
>
> The attached works fine in-house but I really want to clarify how these
> cases should be resolved w/o breaking the protocol.

---

From: Toyo Abe <[email-masked]>
Date: Mon, 14 Dec 2009 11:43:47 +0900

Johan,

Thank you for the comments.
I'm sorry for the late response. I was out of the office last week.

> running as well as the "heartbeat" - CONTACT messages interchange -.  

Yes, I agree.

> transit to next state, CONFLICT_DONE.

It sounds much better than mine.

> recover and the connection is unstable.

I'm not sure if both servers should restart all over again in this case.
It seems somewhat overdoing, though. Or, is it due to a flaw in the state machine
designed in the spec?

> might be better asked on the ISC mailing list.

Umm.. that's my headache.
There seem to be few people in ISC mailing list who are interested in the problem.

Best Regards,
-toyo

---

From: Johan Bosaeus <[email-masked]>
Date: Mon, 14 Dec 2009 11:23:45 +0100

Hello,
On Monday 14 December 2009 03:43:47 am Toyo Abe wrote:
> > We have had the same problems with the spec. Indeed, I would like to
> > widen the problem. You cannot be sure that the UPDREQ command actually
> > reaches the SECONDARY and gets processed there although the TCP
> > connection is up and running as well as the "heartbeat" - CONTACT
> > messages interchange -.
>
> Yes, I agree.
>
> > In this case, we simply resend the UPDREQ command each time we are
> > transiting to the POTENTIAL_CONFLICT state.  The receiving SECONDARY in
> > this case is resilient to multiple UPDREQ:s. The important thing is that
> > the PRIMARY finally obtains the UPDDONE command from the SECONDARY, so it
> > is able to transit to next state, CONFLICT_DONE.
>
> > Same thing here.  You cannot assume  whether the previously sent UPDREQ
> > has been received or not. Sure, the UPDREQ has been received it this
> > case, but in a state, CONFLICT_DONE, which the implementation fails to
> > remember. The draft doesn't specify it either, so I guess the intention
> > of the draft is that both server's should start all over again in
> > POTENTIAL_CONFLICT state.  If so, we may end up in an almost infinite
> > loop when there are *lots* of bindings to recover and the connection is
> > unstable.
>
> It seems somewhat overdoing, though. Or, is it due to a flaw in the state
> machine designed in the spec?
>
Yes,  I think you are right. First of all, the draft mandates that the primary 
should stay in CONFLICT_DONE, even if the connection goes down. On another 
hand, if both servers restarted in POTENTIAL_CONFLICT, the infinite loop will 
only occur if the SECONDARY doesn't get any BNDACK prior to experiencing a 
new disconnect.  In other words, the UPDREQ request sent by the PRIMARY, 
informs the SECONDARY to send BNDUPDs for all bindings the PRIMARY doesn't 
know the current state of, so as the SECONDARY receives BNDACKs, next UPDREQ 
operation will have less to do.

Anyway, what do you think about extending the protocol with a new command, 
UPDREQ_ACK? The recepient of an UPDREQ/UPDREQALL would be obliged to send 
this command to its peer, provided the statemachine is in the proper state, 
to confirm that it is now ready to accept BNDUPD:s for recovery. Then the 
sender knows, upon receiving the UPDREQ_ACK, that it doesn't have to resend 
the UPDREQ command.
Even if our implementation is resilient to reception of multiple UPDREQ's, I 
think it would be even better if we had protocol support for this problem.
 
> > At any rate, the draft standard you refer to was never accepted, so the
> > issue might be better asked on the ISC mailing list.
>
      Best Regards,  
           Johan
> _______________________________________________
> dhcwg mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dhcwg

---

From: Toyo Abe <[email-masked]>
Date: Wed, 16 Dec 2009 16:01:36 +0900

Hello Johan,

>> It sounds much better than mine.

I'd like to clarify this point a bit more. Is resending UPDREQ/UPDREQALL prohibited/illegal in the protocol? or is there any side-effects by doing so?

I'm taking a look at source code of the ISC implementation again. ISC-DHCP doesn't allow to send UPDREQ/UPDREQALL when one is already on the fly, except that UPDREQALL can be sent if on-the-flying request is UPDREQ(they call it 'upgrade').
OTOH, the receiver discards older request when it gets another UPDREQ/UPDREQALL.
So the receiver seems to be ready for multiple UPDREQs. I don't find out the reason why ISC guys disabled multiple UPDREQs to be sent. It may be only a performance reason, though.

> think it would be even better if we had protocol support for this problem.

Although it maybe a good idea, the draft update will be required when ISC
implementation supports the new command. So it'd be very difficult to take
such an approach for my problem.

Thank you,
-toyo

---

From: Johan Bosaeus <[email-masked]>
Date: Wed, 16 Dec 2009 08:35:57 +0100

> I'd like to clarify this point a bit more. Is resending UPDREQ/UPDREQALL
> prohibited/illegal in the protocol? or is there any side-effects by doing
> so?
>
> I'm taking a look at source code of the ISC implementation again. ISC-DHCP
> doesn't allow to send UPDREQ/UPDREQALL when one is already on the fly,
> except that UPDREQALL can be sent if on-the-flying request is UPDREQ(they
> call it 'upgrade'). OTOH, the receiver discards older request when it gets
> another UPDREQ/UPDREQALL. So the receiver seems to be ready for multiple
> UPDREQs. I don't find out the reason why ISC guys disabled multiple UPDREQs
> to be sent. It may be only a performance reason, though.
>
For each new UPDREQ you receive, you must update the list of bindings you are 
going to send to your peer. I.e. every BNDACK received will decrease the size 
of this list. UPDREQALL is less stateful. It simply tells the server to 
resend everything, regardless of what happens. So it is more robust. 
Disallowing multiple UPDREQ:s seems to be calling for problems though, if you 
want a robust POTENTIAL_CONFLICT interchange?
 
Regards,
  Johan

---

From: Toyo Abe <[email-masked]>
Date: Wed, 16 Dec 2009 18:12:52 +0900

> resend everything, regardless of what happens. So it is more robust.

That's the way ISC-DHCP does. I used 'discard', which comes from the comments
in the source code, but it's not appropriate word here.

> want a robust POTENTIAL_CONFLICT interchange?

I might misunderstand what you're asking. If the following comments doesn't meet
your point, I would apologize.

I'm thinking that allowing multiple UPDREQs is a good solution unless it breaks the protocol.
This problem is caused by both servers waiting for the command from its peer.
To avoid the phenomenon, it seems to be a solution to send UPDREQ again to restart the recovery process.
Our customer wants to use DHCP-FO and they found this problem on their lab.
I want to resolve it but cannot accept big change against the function itself defined in the spec.
So I'm asking if the multiple UPDREQs is acceptable from a point of view of the protocol.

Thank you,
-toyo

---

From: Johan Bosaeus <[email-masked]>
Date: Wed, 16 Dec 2009 11:39:24 +0100

>
> > Disallowing multiple UPDREQ:s seems to be calling for problems though, if
> > you want a robust POTENTIAL_CONFLICT interchange?
>
> I might misunderstand what you're asking. If the following comments doesn't
> meet your point, I would apologize.
>
> I'm thinking that allowing multiple UPDREQs is a good solution unless it
> breaks the protocol. This problem is caused by both servers waiting for the
> command from its peer. To avoid the phenomenon, it seems to be a solution
> to send UPDREQ again to restart the recovery process. Our customer wants to
> use DHCP-FO and they found this problem on their lab. I want to resolve it
> but cannot accept big change against the function itself defined in the
> spec. So I'm asking if the multiple UPDREQs is acceptable from a point of
> view of the protocol.
>
> Thank you,
> -toyo

Hmm, my understanding of the draft is that it is not mentioned anywhere that 
resending UPDREQs is something that you should do. On the other hand, I don't 
think the authors of the draft has anticipated this problem fully. So it 
seems like the draft has to be updated, in one way or the other, to meet the 
requirements of a robust exchange of binding updates. As I mentioned before, 
I would actually prefer a more 'lock-step' solution with an UPDREQ/UPDREQ_ACK 
interchange.
      Regards,
         Johan

---

From: Thomas Narten <[email-masked]>
Date: Mon, 21 Dec 2009 09:20:32 -0500

Quick question: what is the motivation for spending significant time
with this protocol?

This is an old document and one that the WG eventually abandoned,
mainly because there really didn't seem to be any interest in the
community in having a solution in this space.

Has that changed? What is the motivatoin for spending time getting
inot some of the finer points of this document given its status?

Note: it doesn't surprise me that there might be serious
defficiencies with the spec... it was abandoned after all because it
was known to not be ready for publication and it didn't seem like
there was energy to finish the work...

Thomas

---

From: Kim Kinnear <[email-masked]>
Date: Mon, 21 Dec 2009 10:20:27 -0500

>community in having a solution in this space.

        The plan from a year or so ago (or maybe more) was to try
        to produce a generic informational draft with specific
        implementation specific informational drafts showing the
        details peculiar to the various similar but distinct
        implementations.

        These drafts have not yet been written, and given the
        level of resources available for "nice to have" as
        opposed to "absolutely required" standards activity in
        today's economy, probably won't happen anytime soon, if
        ever.

>inot some of the finer points of this document given its status?

        Perhaps that the ideas explored in the draft(s) are both
        innovative and have been implemented in a variety of systems
        deployed in the real world.

>there was energy to finish the work...

        My take on this was that the energy available to review
        and approve a standards track document of greater than
        120 pages was simply not there, despite considerable
        effort to generate such interest.  This lack of energy
        was due in part to the complexity and length of the
        document (and protocol), and was coupled with a
        perception that the number of folks who really cared
        about this was (and remains) small.  Which is not
        in conflict with your statement above.

        Since the world (even the DHCP world) has not ended due
        to the lack of a standards track DHCP failover protocol,
        I suppose we can conclude (somewhat reluctantly, on my
        part) that the result of all of this was at least
        acceptable.

        Regards -- Kim

---

From: [email-masked]
Date: Mon, 21 Dec 2009 16:35:12 +0100 (CET)

> community in having a solution in this space.

I'd say there is definitely interest in the community - but mainly in
the sense that ISC, Nominum etc implementations are actually *used*
(in a redundancy setup).

We use ISC DHCP, and are reasonably happy with that version. As long
as it keeps working well for us, the interest in having a sufficiently
detailed/worked through spec that (for example) ISC DHCP and Nominum
could interoperate is only moderate. However, the interest in having
reasonable faith in the protocol *correctness* is still high.

Steinar Haug, Nethelp consulting, [email-masked]

---

From: Bud Millwood <[email-masked]>
Date: Mon, 21 Dec 2009 17:56:48 +0100

> reasonable faith in the protocol *correctness* is still high.

Yeah, I think it boils down to that - correctness.

Johan spent a couple of years creating active/active failover implementations 
for DHCPv4 and DHCPv6, and the failover draft was a very good starting point 
for his work. The authors of that draft did a great job, and we're happy to 
share our experience with this draft back with the DHCP community.

But since this is not strictly for interoperability we can take this 
discussion off-list if that's preferable.

- Bud

Bud Millwood
Weird Solutions, Inc.
http://www.weird-solutions.com
tel: +46 8 758 3700
fax: +46 8 758 3687

---

From: Toyo Abe <[email-masked]>
Date: Tue, 22 Dec 2009 14:02:14 +0900

Hello,

> Quick question: what is the motivation for spending significant time

I think I had already answered your question.
We use ISC DHCP and need DHCP-FO feature due to the demand of our customer.
I asked on ISC mailing lists at first, but there were very few response.
So I raised an issue to the WG, expecting an opinion from the dhcp experts.

> community in having a solution in this space.

Maybe yes. Sorry, I don't know the exact story about it.
But I know there are actually users who have interests in the protocol.

> there was energy to finish the work...

All I can tell about the rest of your question is that I don't have
off-list discussion nor consensus about this. I was just expecting
an opinion from the experts.
So the status you mentioned has not changed, I suppose.

Regards,
-toyo