Subject: [nfsv4] stable storage for server restart

From: [email-masked]
Date: Sat, 19 Feb 2005 18:49:16 -0500 (EST)

Here are a few recollections w.r.t. stable storage for server restart, from
when I did it for my BSD server, in case it is of interest.
(I cc'd [email-masked], just in case anyone not on the linux list is
interested. Apologies in advance for cluttering up your email. Also, I'd be
interested in hearing any thoughts others have on the design.)

Here is a brief description of my design:
- I used a single file written as an append log as follows:
  - at server startup (before any Compounds are performed), the log is
    read and an in-memory structure is created, indicating what clients
    can reclaim state
  - during reclaim, the in-memory structure is used to check for Grace and
    is marked for successful reclaims, per client
* - at the end of the grace period, the file is truncated to 0 length and
    a new append log is written from the in-memory structure, with one record
    for each client that successfully reclaimed some state
  - then normal, non-grace operation starts...
    - records are appended to the log when a client acquires the first state
      (first Open) after a SetClientID and when state is revoked for a
      client (I do not support revocation of only some state for a client)
      (nb: The first Open records are only done for clients that didn't
       successfully reclaim during grace.)

Notes:
- I had to lock the other nfsd threads out when updating stable storage. The
  reason was:
  - I needed to record the revocation before issuing conflicting lock state,
    and I wanted to avoid races between multiple clients trying to acquire
    conflicting locks while the write(s) to disk were in progress. Since
    revocation is a rare event, I didn't see this as a serious performance
    hit.
  - For the case of first Open, the record indicates successful lock
    state acquisition. If another client acquires a conflicting lock
    while the disk write(s) for the log are in progress, there would be
    a record indicating that the client had successfully acquired state
    although the lock failed, due to a conflict. Is this actually a
    problem? I'm not convinced it is, but my code "plays it safe" for now.
    I could see this being a significant performance hit, if lots of new
    clients did SetClientIDs followed by Opens at the same time. (Ones
    that haven't already reclaimed locks at server restart.)
  - The other two cases (when server first starts up and at end-of-grace)
    only occur once per server reboot and only add a little time to the
    grace period.

I think the weakest part of this design is that, if the server crashes
again while at "*", the append only log is not complete (possibly empty).
This will result in clients not being allowed to reclaim, that otherwise
should be able to (ie. no entry->no reclaim->NFS4ERR_NOGRACE for all reclaim
requests).

The append log will grow, but I only see a problem if clients go hogwild
with SetClientIDs. It does get truncated when the server restarts, so a
sysadmin can just reboot when it gets too big:-)

It also doesn't support the notion of only some state for a client being
revoked. (I've looked at that one a bit and it seems to get quite
challenging. Maybe someday I'll come up with a simple scheme I'm convinced
works for that case.)

rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Mon, 21 Feb 2005 14:32:17 -0500

> interested in hearing any thoughts others have on the design.)

i'm also interested in thoughts on our design for the linux server which is 
similar.

instead of appending a recovery file we use the 'file system as a data base' 
approach, populating a (configurable) recovery directory with one directory 
per clientid. the clientid directory name is the md5 hash of the client 
supplied client id which can be of length 1024. the md5 hash is calculated at 
SETCLIENTID, and we return CLIENTID_IN_USE for md5 cache hits, which should be 
negligible. no upcalls are involed, all operations are done in-kernel.
 
>     read and an in-memory structure is created, indicating what clients

we read the recovery directory, populating in-memory structures indicating 
which clients can reclaim state.

>     is marked for successful reclaims, per client

we do the same.

>     for each client that successfully reclaimed some state

at the end of the grace period, we remove the clientid directories from the 
recovery directory for those clients that did not reclaim state.

>        successfully reclaim during grace.) 

normal, non-grace operation starts:
  - we add a clientid directory to the recovery directory when a client makes 
their first successful open confirm. we remove a clientid directory from the 
list whenever either its lease expires or admin action removes the client 
state.

> - I had to lock the other nfsd threads out when updating stable storage. The

for us this is all auto-magically handled by exitsing directory operations 
(mkdir,rmdir), and our nfs state lock.

> I think the weakest part of this design is that, if the server crashes again
> while at "*", the append only log is not complete (possibly empty). This will
> result in clients not being allowed to reclaim, that otherwise should be able
> to (ie. no entry->no reclaim->NFS4ERR_NOGRACE for all reclaim requests).

when we crash at *, our recovery directory has all the clients that have just 
successfully reclaimed state, plus potentially some that could have reclaimed 
state, but didn't during the last grace period. so, i feel ok about crashing 
at * and recovering with the data in the recovery directory.

> The append log will grow, but I only see a problem if clients go hogwild with
> SetClientIDs. It does get truncated when the server restarts, so a sysadmin
> can just reboot when it gets too big:-)

our recovery directory only holds active clients; this is strength of our 
design.

> challenging. Maybe someday I'll come up with a simple scheme I'm convinced

another strength of this design is that the clientid direcories can easily be 
populated with files containg additional info. for example, we plan on adding 
a file to hold SETCLIENTID principal info.

-->Andy

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 22 Feb 2005 00:28:15 -0800

On Sat, [email-masked] wrote:
<...>
> sysadmin can just reboot when it gets too big:-)

The Solaris client will SETCLIENTID with a "fresh" server.
This will happen often if the client unmounts everything from
a server then remounts (new SETCLIENTID dance).  The use of the
automounter may trigger this behavior often; note that automounter
timeouts are on the order of 5/10 minutes.

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 22 Feb 2005 00:39:37 -0800

I wont' go through point by point for Solaris but the approach
is similar to what Andy describes for Linux.  The Solaris server
has a single directory with a file entryfor each clientid.
The filenames are ip-address/clientid with the contents being
the full client supplied clientid.  This offers a "quick" way
to determine which clients are active with a server.

The same type of "move state directory on server recovery" methodology
is used and the methodology will survive the cascading failure issue.

Spencer

On Mon, William A.(Andy) Adamson wrote:
> NFSv4 mailing list
> [email-masked]
> http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4