Subject: [dhcwg] Discussion on draft-ietf-dhc-failover-12.txt

From: "David W. Hankins" <[email-masked]>
Date: Mon, 24 Jan 2005 16:46:41 -0800

Some of you have seen a long and wordy email I sent to the authors of
the draft, at the time I wasn't sure about the status of the draft, in
particular if [email-masked] was really a proper place to ask.

It seems we would at least like to know how much interest there was for
this within the WG still.

It was a very long message, and as I look back possibly needlessly so.
I'll try to summarize this time, and provide summaries of the
answers I've received.

 Q: Why isn't POOLREQ bidirectional?

In the original mail, I had a wordy explanation that it seemed that
ISC DHCP did use POOLREQ bidirectionally once upon a time, but that
this was reverted - I recently fixed a bug that may have made life
in that reality very difficult, and I was wondering if this should
be tried again.

	My input:  It would be much cleaner, codewise, if the secondary
		were the only one to touch BACKUP leases once they
		were handed over...the primary tries to avoid leases
		that are 'likely' to be allocated, but in reality
		it comes down to the fact that it just doesn't know
		what leases are available.

Steve Gonczi's answers (sorry I'm paraphrasing, feel free to correct
me):
	A1: That's a leftover from the start of the draft, it's not
		a bad idea for primaries to send POOLREQs.

My idea in followup:
	A2: We could do away with POOLREQ entirely and have each
		server allocate leases to the other when it detects
		a misbalance in that direction.  Servers check for
		pool misbalances anyway...they may as well run their
		own routine which a POOLREQ would normally excite.
		It's the same math in reverse.  So once both servers
		are POOLREQing away, it suddenly seems like a needless
		complication.

 Q: Why can't the primary return a lease to BACKUP after EXPIRED?

The draft explicitly says it must not be done.

One of the #1 complaints about failover from our users is related to
clients not getting their last lease.  Yes, this behaviour is optional
in rfc2131, it is also highly desirable.

The draft seeks only to address 'client-lease affinity' (lack of
better words) between the servers at POOLREQ time, which I estimate
to be insufficient (it should be noted that at present ISC DHCP
implements 'random client-lease affinity' in that it makes no
attempt whatsoever).

	My input:  The draft is wrong.  Sure, allocating leases to the
		BACKUP state may misbalance the pool in the secondary's
		direction, but allocating *ALL* leases to the FREE
		state definitely misbalances the pool in the primary's
		direction, and for no perceived benefit.  The LB
		algorithm works, it divides the load consistently in
		half, therefore expiring leases will also be
		approximately divided in half.  Each pool should grow at
		approximately the same rate.

Steve Gonczi's answers:
	A1: We should segregate the notion of lease ownership from lease
		state, via a separate attribute.  An expired lease is
		retained by the owner when it comes time to expire, rather
		than returning to any one server.
	A2: "Eager update" instead of lazy update:

		> BTW. I have been thinking about an entirely different
		> approach to Failover: instead of lazy updates, I would
		> consider using "eager updates".

		> From your operational experience, do you think that
		> requiring primary and backup to be connected by a
		> fast link is sane ?

		For a subset of users, yes.  This is in fact what at
		least one user I can think of does already.

		Understand that I think a larger number of users place
		each server in far distant locations...it is actually
		more requested that an additional LB algorithm be
		created, one that would prefer a server by locality,
		rather than dividing client load.  Obviously, I can't
		begin to conceive of how this could be done effectively
		with today's protocols, other than simply causing both
		servers to answer.

		> Rephrased: is there enough deployments out there
		> with a fast connected primary-secondary to warrant
		> another failover protocol optimized for this?

		Possibly.  I think many people read 'failover' and they
		dream of something different, something people would
		actually call 'High Availability'.

		I think if you defined 'fast connected' as a direct
		crossover between the two, then no.  On the other hand,
		servers that are connected to the same switch are a dime
		a dozen.

		I also think the people who would have a single pair of
		DHCP servers with a dedicated link between them would
		have a rather significant client load.  This means what
		they're more interested in is removing bottlenecks (such
		as sync'ing lease state information to persistent storage)
		rather than ease of use.

		If you had two servers 'eager updating' between each other,
		and they had to write data to persistent store whenever one
		updated the other, then what is the functional difference
		between that, and the original failover draft that held one
		server offline while the other answered all queries?  The
		bottleneck is still the slowest drive in the system.

An idea I had in followup:
	A3: We could implement an additional lease state between ACTIVE and
		EXPIRED to indicate the lease is INACTIVE, but still
		available to be allocated to the lease's previous owner.
		Either server could allocate to this single owner alone
		at any time (subject to LB as usual), and since only one
		owner is a valid candidate, there's no risk for duplicate
		allocations.  The primary would judge wether or not the
		FREE/BACKUP pools were wanting, and initiate moving leases
		from here to EXPIRED state then on as usual.

 Q: Why make rebalance checks whenever pool sizes fluctuate?

The draft currently recommends that the pools be evaluated and POOLREQ
like activity commence "whenever the number of available addresses for
either the primary or secondary changes".  Currently, ISC DHCP does
this in a buggy way: whenever a lease is allocated successfully for a
new client.  This obviously doesn't help when there are no leases left
to allocate.  When I seek to fix this, I would like to do it once.

	My input:  If we consistently attempt to judge the pool state
		whenever a pool size decreases, we will need to enter
		sophistication in order to avoid sending multiple POOLREQs
		(or similar) just because we allocated multiple leases
		in a short timeframe.  It may be better to ask the
		servers to perform pool balance checks on an interval
		instead - more frequently for very small pools, less
		frequently for very large pools.

-- 
David W. Hankins		"If you don't do it right the first time,
Software Engineer			you'll just have to do it again."
Internet Systems Consortium, Inc.		-- Jack T. Hankins
_______________________________________________
dhcwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dhcwg

---

From: Ted Lemon <[email-masked]>
Date: Thu, 27 Jan 2005 14:53:06 -0700

> this within the WG still.
I would very much like to see this draft advance, as I'm sure would a 
number of other people.

> in that reality very difficult, and I was wondering if this should

The reason for doing POOLREQ in one direction and lease stealing in the 
other, in my book, is that having two mechanisms is more robust in some 
ways than having one.   I think lease stealing works reasonably well - 
there's no failure mode - so it should be okay to do it.   You are 
right that it would make a lot of sense to do symmetrical POOLREQs, 
though, and in fact that is what I implemented before I had read that 
section of the draft carefully.   My main reason for not pushing on 
this is that the specified mechanism works and has some advantages, and 
I didn't want to make gratuitous changes to the draft because it's been 
so hard to advance as it is.

> The draft explicitly says it must not be done.

We (that is, the DHCP geeks at Nominum) agree that this is a mistake.   
It would indeed be good to return leases to the server most likely to 
reallocate them.

> in rfc2131, it is also highly desirable.
You can make this behavior happen and still follow the current draft, 
but I agree that it would be better if the lease wound up in the right 
place in the first place.

>         than returning to any one server.
We shouldn't make a change like this.   It's too expensive, and doesn't 
give enough benefit.

>         with today's protocols, other than simply causing both
This is easy.   You just set up two failover relationships, and make 
the server that's primary answer 100% of the clients.   Make the server 
that's closest to a client primary for the leases it serves.   I'm 
pretty sure it's possible to have a setup like this with failover, but 
maybe I'm mistaken.   If I am, a second way to do it is to have two 
failover servers at each site, with the primary at each site doing 100% 
of the leases for clients close to that site, and the secondary at that 
site backing up the primary at the other site.   That way you get the 
same effect, albeit at the cost of deploying two extra boxes.

>         bottleneck is still the slowest drive in the system.

For the love of god, let's not redesign the failover protocol.   My 
experience of this is that, considering the example of the ISC DHCP 
server, there's no bottleneck in the protocol that's slowing the server 
down.  The last thing you should be thinking about is redesigning the 
protocol to accommodate the ISC server.   AFAIK Cisco has no 
performance problem with the protocol itself, nor does Nominum.

>         from here to EXPIRED state then on as usual.
This would work, but is it solving a real problem?

>         frequently for very large pools.

The draft doesn't forbid doing this.   I think (based on my own 
experimentation) that you will find that there is no way to choose the 
balance check frequency in such a way that it does not cause problems.

Bottom line: we need to be very careful about making changes to the 
spec unless there's a bug in the spec.  Otherwise it will literally 
never make it to RFC.   Since you and Steve are both primarily 
experienced in the ISC server, I think a lot of the problems you're 
seeing aren't problems in the protocol itself, but rather interactions 
between the protocol and the implementation, and that you should be 
very careful to distinguish between the two.   My experience with DHCP 
performance tuning is that a lot of times the place where it looks like 
the problem lives isn't even related to where the problem actually 
lives.

---

From: "Bernie Volz" <[email-masked]>
Date: Fri, 28 Jan 2005 09:58:18 -0500

While there are always enhancements to be made to anything, I would rather
we focus on getting the existing draft to RFC status. Once that is done, if
there are further improvements to be made, there's nothing stopping that
from occuring.

So, if you have issues that are fundamental flaws, we need to address and
fix those.

Note that technically moving a lease from EXPIRE to BACKUP is allowed
depending on which part of the failover document you read. Figure 7.1.3-1,
"Accepting BNDUPD messages", does allow this since it has "accept" in the
column under "FREE/BACKUP":

                        binding-status in received BNDUPD
       binding-status
       in receiving                                  FREE       RESET
       server          ACTIVE   EXPIRED   RELEASED   BACKUP   ABANDONED

       ACTIVE          accept(5) time(2)   time(1)    time(2)   accept
       EXPIRED         time(1)   accept    accept     accept    accept
       RELEASED        time(1)   time(1)   accept     accept    accept
       FREE/BACKUP     accept    accept    accept     accept    accept
       RESET           time(3)   accept    accept     accept    accept
       ABANDONED       reject(4) reject(4) reject(4)  reject(4) accept

However, Figure 5.11-1, "Transitions between binding-status values" doesn't
allow this.

So, we should clean up this conflict in a future revision of the draft.

Regarding the POOLREQ messages, I see major flaws in allowing both servers
to initiate this. What if both do it at the "same" time. Now, you've got
rebalancing running on both and each server initiates moving some leases to
the other (and likely different leases!). But, once those BNDUPDs are
received, the servers will once again be out of balance and hence another
POOLREQ might be generated (by both) ... So, you say, OK if a server
receives a POOLREQ from the partner after it sent a POOLREQ it "ignores" it
... But now both servers could "ignore" it and then you'd have no
rebalancing. I think eventually you're back to something that requires one
server (main or backup) to do something special. So, why not just avoid all
this and allow only one server to do the POOLREQ?

Regarding the issue of "keeping leases" on the server likely to allocate
them, especially when load balancing (RFC 3074) is being used, this gets to
be extremely tricky as:
A) you never know if a client will return and when it will.
B) you never know if the failover partner will be "up" if the client
returns.
C) you need to keep the lease pools balanced - so, yes, the servers could
move around leases that have no client associated with them, but eventually
all leases are likely to have an associated (old) client and then you've got
to move some.

A server is always free to chose whatever algorithm it wants to rebalance
the pool and move leases around (if an expired lease goes to the main, the
main can give it back to the backup if the client associated with the lease
is likely to be serviced by the backup). Yes, this may involve one more
BNDUPD/BNDACK.

- Bernie

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of David W. Hankins
> Sent: Monday, January 24, 2005 7:47 PM
> To: [email-masked]
> Subject: [dhcwg] Discussion on draft-ietf-dhc-failover-12.txt
> 
> 
> Some of you have seen a long and wordy email I sent to the 
> authors of the draft, at the time I wasn't sure about the 
> status of the draft, in particular if [email-masked] was 
> really a proper place to ask.
> 
> It seems we would at least like to know how much interest 
> there was for this within the WG still.
> 
> It was a very long message, and as I look back possibly 
> needlessly so. I'll try to summarize this time, and provide 
> summaries of the answers I've received.
> 
> 
> In the original mail, I had a wordy explanation that it 
> seemed that ISC DHCP did use POOLREQ bidirectionally once 
> upon a time, but that this was reverted - I recently fixed a 
> bug that may have made life in that reality very difficult, 
> and I was wondering if this should be tried again.
> 
> One of the #1 complaints about failover from our users is 
> related to clients not getting their last lease.  Yes, this 
> behaviour is optional in rfc2131, it is also highly desirable.
> 
> The draft seeks only to address 'client-lease affinity' (lack 
> of better words) between the servers at POOLREQ time, which I 
> estimate to be insufficient (it should be noted that at 
> present ISC DHCP implements 'random client-lease affinity' in 
> that it makes no attempt whatsoever).
> 
> The draft currently recommends that the pools be evaluated 
> and POOLREQ like activity commence "whenever the number of 
> available addresses for either the primary or secondary 
> changes".  Currently, ISC DHCP does this in a buggy way: 
> whenever a lease is allocated successfully for a new client.  
> This obviously doesn't help when there are no leases left to 
> allocate.  When I seek to fix this, I would like to do it once.
>

---

From: "George C. Kaplan" <[email-masked]>
Date: Fri, 28 Jan 2005 09:33:04 -0800

In message <[email-masked]>, Ted Lemon writes
> I would very much like to see this draft advance, as I'm sure would a 

I'm too new to using failover to make any detailed comments on the 
draft, but I want to echo this sentiment.  At our site, 
we're currently doing a long-overdue upgrade of our DHCP servers.  We 
need to use failover, and we'd REALLY like it to be a standard protocol.

> >      [...]

They're a dime a dozen because it's easy to set up.  ("OK, we got these 
two servers; let's put 'em in that rack over there".)  This is fine for 
protecting against a server failure, but it does nothing for you in 
case of a *facility* failure.  We've had enough of those (power 
failures, flooding) that geographical separation between the DHCP 
servers is more important to us than a fast connection between them.

(Before you ask, we're also upgrading the facilities, so that flooding 
and power failures will be less likely).

> protocol to accommodate the ISC server.   

Amen to that.  My understanding is that the failover protocol mostly 
works, and there's a lot of practical experience with it.  Let's move 
ahead with what we have now.

If people think that "eager updates" will help solve a problem, then they
can be explored as a possible future extension.

-- 
George C. Kaplan                            [email-masked]
Communication & Network Services            510-643-0496
University of California at Berkeley

---

From: "David W. Hankins" <[email-masked]>
Date: Fri, 28 Jan 2005 11:09:09 -0800

On Thu, Jan 27, 2005 at 02:53:06PM -0700, Ted Lemon wrote:
> ways than having one.

Please expand?  You and I appear to have completely orthogonal viewpoints
on this matter (in the absence of further discussion on this last week, I
wrote code this morning that effectively makes the ISC DHCP server stop
sending POOLREQ entirely, except as a reverse-compatibility measure).

I see POOLREQ as completely superfluous.

> so hard to advance as it is.

I may be a naive newcomer, but I am unable to accept the time it has
taken to advance a draft as an excuse to forgive its shortcomings.

The time the draft has spent to mature is insignificant compared to the
amount of time it will spend as an RFC.

So please expand on these advantages I am not seeing?

> We shouldn't make a change like this.   It's too expensive, and doesn't 

I have to agree.

Due to this scenario:

 - A client obtains a lease from the secondary when the primary was unable
   to reach the client (for whatever reason).

 - Upon expiry, the lease is returned to the secondary due to its
   'ownership'.

 - Upon the client's return to the network, it gets a new, different lease
   from the primary.

It may be that one server /consistently/ can't reach certain specific
clients, and this method of ownership could seek to remember that.

But that's the only benefit I can imagine at the moment, and the real
fix for this corner case is to solve the communications problem between
the server and client.

> same effect, albeit at the cost of deploying two extra boxes.

Only if you can segregate the clients by subnet (a "shared-network"
doesn't count).  Sorry for not being clear, I didn't think anyone
would write a paragraph that long in response to this anecdote.

> performance problem with the protocol itself, nor does Nominum.

It was not my intention to claim there was a performance bottleneck
described in the current draft.

To restate the part of my comment I think you are responding to;

Eager-updates would require both servers update persistent storage before
any given client received an answer.  This would make the performance
bottleneck the slowest disk in the system.  People who want failover
probably have traffic levels high enough that this would present a
problem for them.

Lazy-updates allows the server to receive multiple BNDUPD's from its
peer, flush them to disk all at once, and respond with BNDACK's at
its leisure.

Consider that people who are able to field "HA" systems also have
large client loads.  It stands to reason that it would be desirable
to divide bottlenecks, and that they would therefore prefer lazy
updates.

Therefore, even if an 'eager-updates' draft was written, people probably
wouldn't use it.

Is that clearer?

> This would work, but is it solving a real problem?

Yes, which is implicit in the above description.

Are you trying to say that this problem is not worth addressing?

> balance check frequency in such a way that it does not cause problems.

It doesn't forbid it, but it makes a statement upon when it should be
done.  If we want to leave this up to the implementors, then so be it,
but the current wording doesn't precisely do that either.

> Since you and Steve are both primarily 
> very careful to distinguish between the two.

I thought that I had distinguished between the two very well.

> My experience with DHCP 

If I ever get around to 'performance tuning', I will be sure to take
this into consideration.

Thank you for your input.

---

From: "David W. Hankins" <[email-masked]>
Date: Fri, 28 Jan 2005 13:15:12 -0800

> So, we should clean up this conflict in a future revision of the draft.

And the light goes on.  Sourcecode has effectively been the first
document I read, and that has caused me to miss the point upon reading
the draft...my understanding of some of the sourcecode was flawed, and
consequently so was my understanding of the draft.

I had imagined a four-way handshake between the servers, moving first to
EXPIRED and then to FREE in two BNDUPD/BNDACK pairs.

I think the draft's documented behaviour is the right one, modulo figure
7.1.3-1 not being clear.

Thanks Bernie.

> this and allow only one server to do the POOLREQ?

The only state that I can imagine the scenario you just described ocurring
is when the two servers are out of synch on lease state information (both
servers would have to believe they have fewer leases than the other).

In which case, both servers would respond by allocating no leases to its
peer (as this would make the misbalance worse from each peer's perspective),
and life would go on.

There is a corner case you don't mention, which /could/ be excited, which
is in the case where the lease misbalance is by only one lease (and the
number of leases available in free and backup is so low that when you are
off by only one lease, you still want to move it).  Each server could
continue moving one lease between each other ad infinitum.

This is easily avoided by implementing your lease rebalance function to
prefer misbalance on one server rather than the other (eg, the pool, if
odd, always gives the primary the extra lease).  Whatever you need to
do to ensure a consistent outcome.

That said, as I mentioned in earlier email, I personally have come to
the conclusion that POOLREQ is entirely superfluous once you start
making both servers do it, and was probably superfluous before that
as well.

- Today, both servers have to check its pool state "now and then" and
calculate the disparity...the primary has to steal leases, the secondary
has to POOLREQ.

- The question of "do I have more or less than my partner" is only a matter
of polarity in the resulting value - the calculation is the same.

- Each server could just as easily allocate its leases to its peer at those
intervals, rather than waiting for the peer to send a POOLREQ.

I have failed to detect a reason to encode this event in the failover
protocol channel.  If one of the two servers was not expected to schedule
a task (wether that be through activity, time, or tea leaves) to check its
lease balance, then I could see the need to remind a peer to do so.

I understand that there was a reason to encode this event in the failover
protocol in the early days of this draft (when LB was not done), as we
would not assume that the servers would work very hard at all to balance
pools, and you may very well need to remind the primary to do so.

So what am I missing?

> is likely to be serviced by the backup). Yes, this may involve one more

...but only in the BACKUP case, as both servers would already have agreed
upon the FREE state upon the EXPIRED BNDUPD/BNDACK.

Which is actually better than what I thought I was asking for.

---

From: "Bernie Volz" <[email-masked]>
Date: Fri, 28 Jan 2005 19:30:11 -0500

> The only state that I can imagine the scenario you just 
> described ocurring is when the two servers are out of synch 
> on lease state information (both servers would have to 
> believe they have fewer leases than the other).

David:

If you have servers that are actively leasing, they'll almost ALWAYS be out
of sync with each other. It takes time to send and process binding updates,
especially if bulking is being used.

Depending on your rate of leasing activity and the size of the subnets
you're using, this may or may not be an issue. But, if the rate is high or
the subnets are small, this can be significant.

Each server can easily keep track of the "FREE" and "BACKUP" lease counts,
but any calculations done on those are suspect because the server only has
an accurate count of its leases -- not its partner's.

In the various servers I've been involved with, we have done dynamic
rebalancing by doing these calculations and moving leases. And, it always
did a relatively good job. This isn't a violation of the failover protocol.
But depending on how aggressive this rebalancing is and when it is done,
POOLREQ are still useful.

- Bernie

> Sent: Friday, January 28, 2005 4:15 PM
> To: Bernie Volz
> Cc: [email-masked]
> Subject: Re: [dhcwg] Discussion on draft-ietf-dhc-failover-12.txt
> 
> 
> > depending on which part of the failover document you read. Figure 
> > 7.1.3-1, "Accepting BNDUPD messages", does allow this since it has 
> > "accept" in the column under "FREE/BACKUP":
> > 
> > However, Figure 5.11-1, "Transitions between binding-status values" 
> > doesn't allow this.
> > 
> And the light goes on.  Sourcecode has effectively been the 
> first document I read, and that has caused me to miss the 
> point upon reading the draft...my understanding of some of 
> the sourcecode was flawed, and consequently so was my 
> understanding of the draft.
> 
> I had imagined a four-way handshake between the servers, 
> moving first to EXPIRED and then to FREE in two BNDUPD/BNDACK pairs.
> 
> I think the draft's documented behaviour is the right one, 
> modulo figure 7.1.3-1 not being clear.
> 
> Thanks Bernie.
> 
> > Regarding the POOLREQ messages, I see major flaws in allowing both 
> > servers to initiate this. What if both do it at the "same" 
> time. Now, 
> > you've got rebalancing running on both and each server initiates 
> > moving some leases to the other (and likely different 
> leases!). But, 
> > once those BNDUPDs are received, the servers will once 
> again be out of 
> > balance and hence another POOLREQ might be generated (by 
> both) ... So, 
> > you say, OK if a server receives a POOLREQ from the partner 
> after it 
> > sent a POOLREQ it "ignores" it ... But now both servers 
> could "ignore" 
> > it and then you'd have no rebalancing. I think eventually 
> you're back 
> > to something that requires one server (main or backup) to 
> do something 
> > special. So, why not just avoid all this and allow only one 
> server to 
> > do the POOLREQ?
> 
> The only state that I can imagine the scenario you just 
> described ocurring is when the two servers are out of synch 
> on lease state information (both servers would have to 
> believe they have fewer leases than the other).
> 
> In which case, both servers would respond by allocating no 
> leases to its peer (as this would make the misbalance worse 
> from each peer's perspective), and life would go on.
> 
> 
> There is a corner case you don't mention, which /could/ be 
> excited, which is in the case where the lease misbalance is 
> by only one lease (and the number of leases available in free 
> and backup is so low that when you are off by only one lease, 
> you still want to move it).  Each server could continue 
> moving one lease between each other ad infinitum.
> 
> This is easily avoided by implementing your lease rebalance 
> function to prefer misbalance on one server rather than the 
> other (eg, the pool, if odd, always gives the primary the 
> extra lease).  Whatever you need to do to ensure a consistent outcome.
> 
> 
> That said, as I mentioned in earlier email, I personally have 
> come to the conclusion that POOLREQ is entirely superfluous 
> once you start making both servers do it, and was probably 
> superfluous before that as well.
> 
> - Today, both servers have to check its pool state "now and 
> then" and calculate the disparity...the primary has to steal 
> leases, the secondary has to POOLREQ.
> 
> - The question of "do I have more or less than my partner" is 
> only a matter of polarity in the resulting value - the 
> calculation is the same.
> 
> - Each server could just as easily allocate its leases to its 
> peer at those intervals, rather than waiting for the peer to 
> send a POOLREQ.
> 
> I have failed to detect a reason to encode this event in the 
> failover protocol channel.  If one of the two servers was not 
> expected to schedule a task (wether that be through activity, 
> time, or tea leaves) to check its lease balance, then I could 
> see the need to remind a peer to do so.
> 
> I understand that there was a reason to encode this event in 
> the failover protocol in the early days of this draft (when 
> LB was not done), as we would not assume that the servers 
> would work very hard at all to balance pools, and you may 
> very well need to remind the primary to do so.
> 
> 
> So what am I missing?
> 
> > A server is always free to chose whatever algorithm it wants to 
> > rebalance the pool and move leases around (if an expired 
> lease goes to 
> > the main, the main can give it back to the backup if the client 
> > associated with the lease is likely to be serviced by the backup). 
> > Yes, this may involve one more BNDUPD/BNDACK.
> 
> ...but only in the BACKUP case, as both servers would already 
> have agreed upon the FREE state upon the EXPIRED BNDUPD/BNDACK.
>

---

From: Ted Lemon <[email-masked]>
Date: Sun, 30 Jan 2005 11:12:49 -0700

> servers would have to believe they have fewer leases than the other).

Just a quick note - this can and frequently does happen if the two 
servers get behind in processing BNDUPD messages.   I think we've tuned 
a lot of this out of our server, but it's a real problem, and I don't 
think it can be completely tuned away.   So it really is something that 
needs to be accounted for.