Subject: [nfsv4] recent server request cache

From: [email-masked]
Date: Sun, 16 Nov 2003 16:59:13 -0500 (EST)

Hi,

I am concerned w.r.t. the possibility of a "false hit" on the server
recent request cache, if a client re-uses an xid (especially just
after a reboot). I've written a brief note on this that is available at
ftp://snowhite.cis.uoguelph.ca/pub/nfsv4/server-cache.notes, in case
anyone is interested.

In summary, I'm curious if there is an accepted practice for selecting
an xid value, especially just after a reboot? (RFC1831 doesn't have
much to say on the matter.)

Thanks for any comments, rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Eric Werme USG <[email-masked]>
Date: Mon, 17 Nov 2003 11:52:05 -0500

What I did ages ago, see http://www.connectathon.org/talks96/werme1.html
was just take the current time and multiply by more than the number of
requests we could do per second.  We can do a lot more now (but usually
don't) and the logic doesn't work all that well with reboots after 100s of
days.  However, the impetus was reboots during hardware and software
maintenance so things work out okay.  I kind of like your idea of the
client writing out the current counter(s) from time to time.

One issue is that the system has an accurate time, but that's pretty
much true with systems running today.

One complication is that on our NUMA systems, I allocate subsets of XIDs
to that various "Resource Affinity Domains" (RADs), groups of CPUs with
equal access to subsets of system memory and keep the XID counter in per-RAD
memory.  Not only does this reduce memory contention, I use it to look
at the right RAD to find the thread to wake when the reply comes back.

While it does reduce the effective width of the XID, at least each RAD is
cycling through XIDs slower than if there were a single counter.

32 bit XIDs are too small these days.  RPC V3 anyone?  Add a client call
to tell the server which DRC entries can be retired, and maybe a count of
the number of requests sent per XID so the server can free DRC space.
Topic for another day?

/*
 * Alloc_xid presents an interface which kernel RPC clients
 * should use to allocate their XIDs.  Its implementation
 * may change over time (for example, to allow sharing of
 * XIDs between the kernel and user-level applications, so
 * all XID allocation should be done by calling alloc_xid().
 *
 * A note on clntxid initialization from 1992:
 * In order to avoid accidentally reusing XIDs over reboots, we try to
 * set clntxid to something higher than what had last been used.  Since
 * we don't do much more than 1000 NFS calls per second, multiplying by
 * 4096 ought to be safe.  That gives us a 12 day window before we wrap
 * around.  Old code tried a lousy attempt as a random number, this has
 * a better, albeit imperfect, chance of being right.  You had better
 * set the time before you call this, hence this is not part of rpcinit().
 *
 * A note on clntxid from 2003:
 * For NUMA support, we're absconding with the high bits of the XID
 * to tuck in what should be the RAD ID of the requesting thread.  So
 * we're reducing the bits available to a particular CPU.  NFS can now
 * support a lot more than 1000 calls per second, but things will
 * still behave reasonably wrt to reuse avoidance.  We use the RAD ID
 * to determine which rad's pending queue to check when replies come in.
 * There are a couple caveats.  Some threads may have migrated to another
 * RAD, so we have to use the XID, not the current RAD to pick the queue
 * to wait on. (nfsiod threads are bound to a RAD so they'll behave fine.)
 * Also, some applications can supply their own XID, and we have to make
 * sure we don't reference a non-existant RAD.  We'll put those on RAD
 * 0, which always exists.
 */

	-Ric Werme
-- 
Eric (Ric) Werme         |  [email-masked]
Hewlett-Packard Co.      |  http://werme.8m.net/