Subject: [dhcwg] Failover (whee)

From: "David W. Hankins" <[email-masked]>
Date: Mon, 17 Jul 2006 10:29:50 -0700

In bringing our implementation up to comply with draft revision 12,
I noticed that the RECOVER-WAIT state, while documented now in
section 9.6, does not appear in the section 12.24 table that gives
these states protocol wire numbers (12.24 also lists 1-9 as a value
for the state, when values at least up to 11 are now documented).

Could the author(s) please ACK on RECOVER-WAIT's protocol number
assignment?  DHCWG's email archive counts as reference for drafts.

While I'm on the subject, I want to describe some changes we're
making that I think are preferrential behaviour (we're releasing
these experimentally, and based upon our findings we hope to
convince the authors to update the draft).

1) "MAC Address Affinity" - It has very little to do with MAC
   addresses, unless the client(s) did not supply client
   identifiers.  Simply put, the primary, upon moving a lease
   into the FREE state for any reason (expired, released), checks
   the LBA for the last client that held the lease.  It transfers
   the lease to BACKUP (queues a new binding update) if the LBA
   hashes to the secondary.  When pool rebalances are iterated,
   there are now potentially two passes: affinity is preserved in
   the first pass, allowing a bit of skewed balance.  Balance is
   preserved in the second pass.

	o It's expected approximately half of the leases that enter
	  the FREE state would be transferred to the secondary, if
	  LBA is doing its job.  This should result in fewer pool
	  rebalance events.
	o Plus, clients are now more likely to get their old address
	  when they return (in the normal operating case).

2) Pool balance calculations are now done on a schedule.  In the
   last version of the software, they were done only when a lease
   was successfully allocated to a client.  This made it rather
   hard to balance pools once they were exhausted.

   Frequent allocations made by the peer move the schedule downwards
   (estimating when the pool will be in need of rebalance), but never
   under a lower threshold.  Rebalance checks always happen at least
   on the upper threshold.

   So to be clear: receiving binding updates from the peer out of the
   peer's "free state" (free or backup accordingly) may reduce the
   schedule depending on how long it's been since the last update.
   These may be back-to-back, particularly at startup, in which case
   the schedule would be brought to the minimum.

   Should either server receive a POOLREQ, it is jarred into
   performing a balance check (POOLREQ then serves as a means
   to ensure the servers don't go blissfully unaware) early,
   and reschedules the next one accordingly.

	o No more excessive calculation in the path of network
	  reaction.
	o No more pool starvation with no way to 'self heal'.

3) No more lease stealing.  The servers are now symmetric, and
   execute precisely the same paths (including MAC affinity as
   applied to lease rebalance events - the behaviour of released
   or expired leases remains a primary server only function).

   Rather than the primary stealing leases from the secondary,
   the secondary gives them away when its own balance calculations
   demand it.

	o No more binding update failures due to missed binding
	  updates or timing.
	o No more 'guessing' what leases the secondary is
	  'unlikely' to allocate.
	o Another vector of failure to 'self heal' is removed.

	o Servers might actually give the peer more addresses than
	  they keep themselves, up to a configured threshold, if
	  the peer matches the previously held client on LBA.

4) No more POOLREQ (well, not really).  Since both servers are
   now in charge of balance calculations, and cooperate to
   maintain it, POOLREQ is a weapon of last resort that is only
   used if the misbalance is more than double the allowed
   threshold.

   It should be underscored here that I'm using POOLREQ
   symmetrically.  Previously, the secondary was the only server
   that produced it.  It's now a panic button for both servers.

	o An older software behaviour, transmitting a POOLREQ
	  every time a client was given an address (and the pool
	  was misbalanced) is no more, since these are only
	  transmitted on the balance-checking scheduled event.

For discussion of protocol enhancement, POOLREQ is a particular
sticking point with me.  DHCP Servers have many pools, each of
which must have balance preserved individually.  Ours, at least,
does not name such pools (in order to distinguish between
syslog lines about differing pools, the original author actually
elected to syslog, in hex, the address of the pool structure in
memory).

Why can't POOLREQ request a balance check be made of an individual
pool?  Right now it cuases all pools in configuration to be checked
in rapid series.  This becomes an increasingly larger problem as
installations grow.

A trivial idea for implementing (since not all platforms name
pools, and not all platforms would name pools in consistent
ways) would be to simply include as an option in the POOLREQ
message an IPv4 address that's a member of the target pool (any
member, in any state).

This can be 'worked backwards from' in precisely the same way
servers work backwards from giaddr (but can also be used in a
more direct way if the server's lease structure in memory gives
it a direct reference to the pool it's a member of).

So instead of rebalancing all pools, you either rebalance all
pools within the shared network of the address provided, or in
the best case the specific single pool (which generally is a
very trivial calculation in comparison).

New subject on protocol enhancement: detecting lease database
skew (rather than leaving it up to the operator).

The most trivial idea building on my suggested POOLREQ change:
produce a new message type that informs the peer of what the
local system believes is the current pool 'state count'.  Of
the form:

	ip-address total free backup active "other"

Where the ip-address, again, is any random member of the pool.

Should the results "substantially" differ from the local system's
idea, in particular: if the primary's count of backup leases differs
from the secondary's count of backup leases, and let us also imagine
that the secondary attempted a POOLREQ but got a POOLRESP indicating
a zero value, then the secondary server receiving such a message MAY
consider sending an UPDREQ or UPDREQALL in order to synchronize the
databases ("I give up, let's start over.").

Which leads me to my final subject:

That UPDREQ/ALL might also use an ipv4 address in an option, to
limit the scope of the UPDREQ/ALL to a specific pool that is in
need of such healing.

-- 
David W. Hankins	"If you don't do it right the first time,
Software Engineer		you'll just have to do it again."
Internet Systems Consortium, Inc.	-- Jack T. Hankins
_______________________________________________
dhcwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/dhcwg

---

From: "David W. Hankins" <[email-masked]>
Date: Mon, 17 Jul 2006 14:23:50 -0700

On Mon, Jul 17, 2006 at 12:01:29PM -0700, Steve Gonczi wrote:
> A few years ago, I posted on the DHC Wg list,
> somewhat dovetailing with the direction I discern from your post.
> 
> I consider it undesirable to cycle leases through the primary,
> as part of the normal lease life cycle.

I think there certainly needs to be a default state to which the
lease returns after it has expired, or released, or reset (and
that it is actually both servers that agree on this through
binding updates, not one server dictating to the other).

I suspect that if we asked both servers to perform an LBA of
the last-client for the lease to determine the next binding
status of the lease, any source of entropy in those servers'
idea of the last client (eg different ideas of who the last
client was, or which identity the client supplied) would
extend into these lease states being inconsistent.

Lease states are the paramount importance of the database - the
cached last client data or any ancillary values stored on
the lease are largely secondary.

So anything we can do to reduce entropy-magnification there is
good.

I don't mind having to do the 'extra step' of transmitting and
ACKing a binding update outside of this system.

But we can talk about short-cutting it: adding an option to the
binding update that indicates the peer's preference for lease
state after the update.  This I would think of as an optimization,
and would have protocol compatibility issues (so we'd have to rev
the protocol-level versioning to introduce it).

> I also believe that lease lifecycle stages should be 
> Distinct and independent of lease ownership status.
> 
> If a lease becomes "free", the server who issued it, 
> Should be able to retain it in its pool(s) unless there is 
> A deliberate action to change the ownership of the lease.
> 
> If we had the above 2 aspects cleanly separated, a lease could be
> transferred from one server to another, regardless of the life cycle 
> stage it is in.
> (the transfer(s), of course could be initiated By either server) 

I think we already have this.

Take your pool of leases, split it precisely in two (or "n" which
for now I will assume is 2).  Run the two servers in parallel with
no state sharing across.  Each own the leases in their range
according to their configuration and regardless of lease state.  You
could easily manufacture a 'configuration channel' that, from a
central authority, shifted ownership between the two servers (you
could even fetch and store lease information between the two as
you shifted these responsibilities).

Of course, if you ever have more active leases than either half
of the pool, you're in trouble.  That's the canonical DHCPv4
problem...its core defining characteristic is presiding over
resource starvation.  We have to assume our pools will run at
80-90% allocated, at the low end.  Many folks I hear from complain
to me because our software doesn't operate quite so well as it
could at 100% allocation (the client id issue, don't get me
started).

Failover's core defining characteristic is that it wants to
operate with at most 2 servers (not n*2 or n+1), entirely for
the systems administrators' expedience ("centralized
infrastructure").

That's a design goal that doesn't resonate with everybody, and
goes in and out of fashion depending on the year.

Once you require two servers to be cognizant of the entire lease
pool range, you need to define strictly four 'ownerships':

	1) Owned by server A, for allocation to new clients (free).
	2) Owned by server B, for allocation to new clients (backup).
	3) Owned by the clients (active).
	4) Not suitable for any other use until it can be agreed upon
		(expired, released, reset, abandoned).
		- But abandoned is even an odder duck that maybe I'll
		  talk about later.

When a failover server loses its partner, it should retain access
to all the (free|backup)+active leases.  That is, it should continue
to be able to extend active leases, and to give new clients addresses
from its reserved portion of the pool.  If it knows for some reason
that its partner is down (or assumes this via some automated means),
then it 'owns' the entire database.

Note carefully my attachment to n=2.  If you were to raise n, you
might acheive substantially more pallateable results.  Where n is
merely 10, you would still have access to 90% of your leases where
one DHCP server had failed, and would "only" ask at most 10% of
your clients to REBIND.

This design goal also does not resonate with everybody, and
also goes in and out of fashion year by year.

> I would also like to see ownership management possibly 
> based on the Load Balancing rfc, or some similar hashing concept,
> e.g. transferring HBA-s to transfer ownership.

I think one possible next-generation might be to define a more
than binary HBA.  EG instead of a 256 bit bucket, a 256 server-slot
assignment, where the contents identify one of the servers in an
array (rather than the binary primary/secondary we have today in
failover).

In such a condition, not only is it unwise to designate 256
distinct lease states, it's infeasible.  You would have to
adopt a separation between state and ownership as you describe,
and that would turn out much easier to implement.

That's a lot of new work.  Not necessarily bad work.

But for Failover's binary relationship, encoding ownership
into lease states is both complete to our needs and far more
expeditious.  It's easier to look and see that the lease is
free, or backup, than to look and see that the lease is free
and then have to consult a table for ownership status.

Even up to let us imagine 4 servers, 4 distinct free-lease
states would be far easier to implement than a separate layer
to make them distinct.

> At the time this was discussed on this list I could not quite 
> Comprehend the objections and reasoning why the separation of 
> Ownership and lease status was a bad idea. It crossed my mind
> that some folks (heaven forbid) may have a shipping product
> based on some variant of the draft, and have to deal with 
> a tested code base, installed customer base etc...
> That _would_ explain reluctance to change anything non-trivial 
> in the Draft. ;-)

I don't think it's a bad idea.  I think it would complicate
some bits and make other bits easier.

I think in particular it would be a good idea if you were to
accept more than two servers as a design goal.

I'll say this for failover's "three vague owners" style: it does
not result in sending clients back to REBIND or even INIT states
in order to pass between primary and backup server(s).

That design goal certainly resonates with most people.

But it's certainly true that today, there are "large" networks
that have products using the failover protocol deployed and
running.

Substantial (non-incremental) changes would send them back to
drawing boards.

Did you notice at the IETF 66 tech plenary that DHCP Failover
was on their summary slide of the network design?

"You're soaking in it."