Subject: RE: [dhcwg] Failover: poolreq message secondary-only?

From: "Bernie Volz (EUD)" <[email-masked]>
Date: Fri, 2 Nov 2001 19:02:34 -0600

The primary can also simply initiate a reclaim of addresses by sending requests to the secondary.

We originally set up the PoolReq the way it is documented before load balancing really seriously entered the picture. But, we did discuss this several times after load balancing was introduced and felt it wasn't necessary since the flow of addresses was typically back to the primary and since it "owned" these, it was more in control of what it gave to the secondary.

Either server can request state changes to move an address to itself.

Note sure if this explains it well enough. But, I don't see a real need for doing this. Ted, perhaps you have a specific case for why this might be needed? It would be interesting to hear of it.

- Bernie

-----Original Message-----
From: steve [mailto:[email-masked]]
Sent: Friday, November 02, 2001 6:23 PM
To: 'Ted Lemon'
Cc: [email-masked]
Subject: RE: [dhcwg] Failover: poolreq message secondary-only?

Given what we have, leases flow to the primary, as they get assigned.

What you propose would allow the primary to reclaim some of the un-used
backup
pool during normal operation.

I believe the current protocol already allows for this during
synchronization.
After the primary received all updates from the secondary, it can safely set
the status of any "backup" lease to "free", because the backup is not
running
until the synchronization is complete.

All said, what you propose is useful and worth doing.

/sg

-----Original Message-----
From: Ted Lemon [mailto:[email-masked]]
Sent: Friday, November 02, 2001 5:32 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [dhcwg] Failover: poolreq message secondary-only?

> Leases should belong to either the primary or the backup,
> regardless of their status ( free, active, abandoned...).
>
> This would be conceptually much cleaner.

Good heavens, that wasn't what I was proposing at all.   That would require
a complete rewrite of the spec, which is out of the question.   I'm just
asking for the restriction that only the secondary can send poolreq, and
the primary send poolresp, be removed.

_______________________________________________
dhcwg mailing list
[email-masked]
http://www1.ietf.org/mailman/listinfo/dhcwg

---

From: Ted Lemon <[email-masked]>
Date: Sun, 4 Nov 2001 17:24:45 -0700

[I accidentally sent this response just to Bernie, which means that
now I get a chance to edit my response to reduce the hot-pepper level.
Sorry about that, Bernie!   :']

It is clear why the draft is done this way - it's because of the history
of failover being a master/slave protocol.   Byt failover is no longer
a master/slave protocol - each asymmetry in the protocol exists for a
specific reason, except for this one.   I would really like this
asymmetry to be fixed, because it's quite clunky - there are two different
mechanisms for pool balancing, where we need only one.

If the secondary is short on leases, it's easy for it to get more - it
  just sends the primary a POOLREQ message.   The primary knows what leases
are free, and sends some of those to the secondary.   This is very
straightforward, and there's no margin for error.

On the other hand, if the primary is short on leases, the primary runs
through the list of leases it thinks are in the backup state, and sends
BNDUPD messages to the secondary moving some of those leases into the
free state.   If the leases on the secondary that the primary chooses
happen to have been assigned to clients, the secondary sends BNDACK
messages with a REJECT_REASON, so that the primary doesn't get the leases.
The primary will continue in this way until it gets enough leases,
presumably, but I hope you can see how much messier this protocol is than
the other.

To evaluate these two mechanisms, just look at the amount of text required
to explain them.   We're always complaining about how long the protocol
specification is, even though that's to some degree unavoidable.   But
here we have a chance to make it smaller, and make implementors' lives
easier.   I think we should take it.

 > Not sure if this explains it well enough. But, I don't see a real need for
 > doing this. Ted, perhaps you have a specific case for why this might be
 > needed? It would be interesting to hear of it.

I have a customer who is using failover actively at her site, with load
balancing.   At her site, it just happens that a decent percentage more
clients hash to the primary than to the secondary.   As far as I can
tell, this is pure chance.   So the primary is always short on leases,
and needs to rebalance the pool, using the clunky primary backup-lease-
stealing method.   I haven't implemented it, because it's complicated.
Implementing the symmetric POOLREQ scheme involved *deleting* a single
line of code - a line of code that tested to see if the server was the
primary, and if it was, refused to send a POOLREQ message.

It doesn't make sense to have language in the draft that complicates
implementation and does not provide a benefit.   The failover protocol draft
has not been issued as an RFC yet, so at least in theory there's still time
to fix this.   I sent another suggestion to the WG that I think also helps
to clean things up (ironically, by *adding* an assymetry), so this would be
a good opportunity for a minor spin, assuming the WG agrees with these 
changes
and Kim has time.

---

From: Ted Lemon <[email-masked]>
Date: Sun, 4 Nov 2001 18:50:21 -0700

Consider this scenario: two servers are running in communications-
interrupted because of a network outage.   The safe period expires,
and both servers make the transition into partner-down.   MCLT expires,
and the secondary reclaims some free addresses to assign, and assigns
one to a client.   The primary is also assigning addresses, although
because of the partition no actual conflicts occur.

The network partition is finally fixed shortly thereafter, and the
servers are able to reestablish communications, at which point
both servers make the transition into the potential-conflict state.
The primary sends an UPDREQ to the secondary, which dumps its lease
database to the primary and sends an UPDDONE.   The primary makes the
transition into conflict-done and begins freely assigning leases.
(By the way, Figure 9.8.3-1 does not reflect the new conflict-done
state.)

Since the network partition has healed, the primary can hear requests
from all networks, and one of the first leases it assigns happens to be
the same free lease that the secondary reclaimed and assigned.   This
creates a needless conflict.   The conflict will be caught during the
secondary's update, but this is cold comfort, since we potentially have
an address conflict at this point.

The point here is that it is not technically safe for the primary to
behave as if it were in the normal state when the secondary is still
in conflict-detected.   It is safe for the primary to make changes to
leases that it's confirmed with the secondary, but until the secondary
makes the transition into normal state, it is not safe for the primary
to assume, for example, that leases it thinks have always been in the
free state are still in the free state.

I would say that the right thing to do is to say that in the conflict-
done state, the primary MAY continue not to service DHCP clients, or
it MAY serve DHCP clients using only those leases that it confirmed
with its peer in the conflict-detected state.

---

From: Kim Kinnear <[email-masked]>
Date: Mon, 05 Nov 2001 11:02:43 -0500

At 06:50 PM 11/4/2001 -0700, Ted Lemon wrote:
>and both servers make the transition into partner-down.

        The theory behind the safe period is to have it be long
        enough so that such a situation never happens.  That is
        to say, that the safe period was *supposed* to be long
        enough that other detection techniques would come into
        play and alert the administrators/operators to the fact
        that there were problems *prior* the safe period
        expiring.

        That doesn't negate your concern, below, but it is
        worth considering in general.

>   MCLT expires,

        I'm a little unclear on this -- did the secondary and
        primary assign the same address, each on its own side of
        the partition?  And "no actual conflicts occur" because
        at the time this happens the clients themselves aren't
        connected?  And when the partition is healed there *will*
        be a conflict?

        No, now I see (from below) that only the secondary
        actually leased the IP address -- right?

>(By the way, Figure 9.8.3-1 does not reflect the new conflict-done

        Thanks -- I'll fix it.  In fact, figure 9.8.3-1 should be
        titled 9.10.3-1 as well.  Sigh.

>the same free lease that the secondary reclaimed and assigned.   

        So I guess that this answers my question above -- the
        secondary took one of the primary's available addresses
        and assigned it, but the primary did not itself actually
        assign that IP address to anyone.

        ---------

        But, I think the primary would not actually assign the IP
        address to anyone after it goes into NORMAL state.  Part
        of the work of the primary processing the secondary's
        lease database is for the primary to resolve conflicts in
        real time while receiving the BNDUPD's.  The secondary
        cannot send the UPDDONE until the primary has processed
        all of the BNDUPD's in some way.

        The primary should receive the BNDUPD from the secondary
        with the leased status in it.  It will show that the IP
        address is currently available, and it will decide to
        honor the leased status from the secondary over its own
        available status.  [You know, rock dulls scissors, no,
        wait, this is leased wins over available.] See table
        7.1.3-1.  It says that the primary should accept the
        BNDUPD from the secondary in this case.

>This

        I think the conflict was caught while processing the BNDUPD
        from the secondary to the primary, so we should probably resolve
        our understanding of this before we move any farther along this
        track of discussion.

>in conflict-detected.  

        I disagree, but the fundamental issue is when the conflict is
        detected and dealt with, so we should resolve that before we
        discuss this.

        Cheers -- Kim

> It is safe for the primary to make changes to

---

From: Ted Lemon <[email-masked]>
Date: Mon, 5 Nov 2001 11:58:40 -0700

>         all of the BNDUPD's in some way.

I misspoke slightly there - I said "NORMAL" when I meant "CONFLICT-DONE."   
The primary will think that the address is in the free state until it gets 
the update from the secondary telling it that the secondary assigned it to 
a client.   So there's a window, potentially fairly long at a large 
installation, in which the primary can allocate that address, thinking it 
is free.   If it does so, the problem will be discovered during the 
resolution process, but it'll be too late at that point - two clients will 
have been assigned the same IP address.