Subject: [nfsv4] CB_RECALL_ANY proposal

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 13 Feb 2006 10:39:33 -0500

This note proposes a generalization of CB_RECALL_ANY, which was first 
proposed as part of the directory delegations draft.  This feature
is useful for multiple types of recallable objects, probably including 
those that haven't yet been thought of, and it would be good if the
version included in v4.1 accommodated all current possible uses as
well as having a good chance of being adaptable to future uses.

The following text is based on Saadia's directory delegation draft.

New Callback Operation: CB_RECALL_ANY - Limit recallable objects

   SYNOPSIS
        N, type_mask -> {}

   ARGUMENT
        struct CB_RECALLANYY4args {
                uint4          objects_to_keep;
                uint4          type_mask;
        }

        const TYPE_MASK_RDATA_DLG = 0x0001;
        const TYPE_MASK_WDATA_DLG = 0x0002;
        const TYPE_MASK_DIR_DLG = 0x0004;
        const TYPE_MASK_LAYOUT = 0x0008;

   RESULT
        struct CB_RECALLANY4res {
               nfsstat4        status;
        };

   DESCRIPTION:
        The server may decide that it cannot hold all of the state for
        recallable objects, such as delegations and layouts, without 
        running out of resources.  In such a case, it is free to recall
        individual objects to reduce the load but this would be far 
        from optimal.

        Because the general purpose of such recallable objects as
        delegations is to eliminate client interaction with the server,
        the server cannot interpret lack of recent use as indicating
        that the object is no longer useful.  The absence of visible
        use may be the result of a large number of potential operations
        eliminated.  In the case of layouts, the layout will be used
        explicitly but the meta-data server does not have direct
        knowledge of such use.

        In order to implement an effective reclaim scheme for such 
        objects, the server's knowledge of available resources must be
        used to determine when objects must be recalled with the 
        clients selecting the actual objects to be returned.

        Server implementations may differ in their resource allocation
        requirements.  For example, one server may share resources among
        all classes of recallable objects whereas another may use
        separate resource pools for layouts and for delegations, or
        further separate resources by types of delegations.

        When a given resource pool is over-utilized, the server can
        issue a CB_RECALL_ANY to clients holding recallable objects of
        the types involved, allowing it to keep a certain number of
        such objects and return any excess.  A mask specifies which 
        types of objects are to be limited.  The client chooses, based
        on its own knowledge of current usefulness, which of the objects
        in that class should be returned.

        When an undefined bit is set in the type mask, NFS4ERR_INVAL
        should be returned.  However even if a client does not support
        an object of the specified type, if the bit is defined, 
        NFS4ERR_INVAL should not be returned.  Future minor versions
        may expand the set of valid type mask bits.

        CB_RECALL_ANY specifies a count of objects that the client may
        keep as opposed to a count that the client must return.  This
        is to avoid potential race between a CB_RECALL_ANY that had a 
        count of objects to free with a set of client-originated 
        operations to return layouts or delegations. As a result of the 
        race, the client and server would have differing ideas as to how 
        many objects to return.  Hence the client could mistakenly free 
        too many.

        The client can choose to return any type of object specified 
        by the mask.  If a server wishes to limit use of objects of a
        specific type, it should only specify that type in the mask
        sent.  The client may not return requested objects and it is 
        up to the server to handle this situation, typically by doing
        specific recalls to properly limit resource usage.  The server 
        should give the client enough time to return objects before 
        proceeding to specific recalls. This time should not be less
        than the lease period.

        Servers are generally free not to give out recallable objects
        when insufficient resources are available.  Note that the 
        effect of such a policy is implicitly to give precedence to
        existing objects relative to requested ones, with the result
        that resources might not be optimally used.  To prevent this, 
        servers are well advised to make the point at which they start 
        issuing CB_RECALL_ANY callbacks somewhat below that at which they
        cease to give out new delegations and layouts.  This allows the 
        client to purge its less-used objects of whatever appropriate and
        so continue to have its subsequent request given new resources
        freed up by object returns.

   ERRORS
        NFS4ERR_RESOURCE
        NFS4ERR_INVAL

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Feb 2006 12:11:09 -0500

>         const TYPE_MASK_LAYOUT = 0x0008;

If the intention is to allow future revisions to increase the number of
masks, would it not be better to make type_mask a full-blown bitmap?

Given that pNFS layouts are likely to differ greatly in size depending
on the layout type a given client has chosen to use, perhaps we might
want to allow for something more fine-grained than TYPE_MASK_LAYOUT?

Say the server suddenly notices that it has underestimated the number of
resources it needs to free up. Is it immediately free to send a new
CB_RECALL_ANY to any given client? If so, how should that client deal
with any races between the 2 CB_RECALL_ANY callbacks?

Note that there exists at least one case where the server is not free to
refuse to give out recallable objects: namely that of a reboot recovery.
However, one can easily imagine situations where the server comes up
again with fewer memory resources than when it crashed. Is there
anything we can do to allow servers to deal with this situation too?

Cheers,
  Trond