Subject: [nfsv4] CB_RECALL_ANY, take 2

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 16 Feb 2006 13:19:00 -0500

This is a second pass, reflecting Trond's comments.  Thanks, Trond.

New Callback Operation: CB_RECALL_ANY - Limit recallable objects

   SYNOPSIS
        N, type_mask -> {}

   ARGUMENT
        struct CB_RECALLANYY4args {
                uint4          objects_to_keep;
                bitmap4        type_mask;
        }

        const TYPE_MASK_RDATA_DLG = 0;
        const TYPE_MASK_WDATA_DLG = 1;
        const TYPE_MASK_DIR_DLG = 2;
        const TYPE_MASK_FILE_LAYOUT = 3;
        const TYPE_MASK_BLK_LAYOUT_MIN = 4;
        const TYPE_MASK_BLK_LAYOUT_MAX = 7;
        const TYPE_MASK_OBJ_LAYOUT_MIN = 8;
        const TYPE_MASK_OBJ_LAYOUT_MAX = 11;
        const TYPE_MASK_OTHER_LAYOUT_MIN = 12;
        const TYPE_MASK_OTHER_LAYOUT_MAX = 15;

   RESULT
        struct CB_RECALLANY4res {
               nfsstat4        status;
        };

   DESCRIPTION:
        The server may decide that it cannot hold all of the state for
        recallable objects, such as delegations and layouts, without 
        running out of resources.  In such a case, it is free to recall
        individual objects to reduce the load but this would be far 
        from optimal.

        Because the general purpose of such recallable objects as
        delegations is to eliminate client interaction with the server,
        the server cannot interpret lack of recent use as indicating
        that the object is no longer useful.  The absence of visible
        use may be the result of a large number of potential operations
        eliminated.  In the case of layouts, the layout will be used
        explicitly but the meta-data server does not have direct
        knowledge of such use.

        In order to implement an effective reclaim scheme for such 
        objects, the server's knowledge of available resources must be
        used to determine when objects must be recalled with the 
        clients selecting the actual objects to be returned.

        Server implementations may differ in their resource allocation
        requirements.  For example, one server may share resources among
        all classes of recallable objects whereas another may use
        separate resource pools for layouts and for delegations, or
        further separate resources by types of delegations.

        When a given resource pool is over-utilized, the server can
        issue a CB_RECALL_ANY to clients holding recallable objects of
        the types involved, allowing it to keep a certain number of
        such objects and return any excess.  A mask specifies which 
        types of objects are to be limited.  The client chooses, based
        on its own knowledge of current usefulness, which of the objects
        in that class should be returned.

        For NFSv4.1, sixteen bits are defined.  For some of these,
ranges
        are defined and it is up to the definition of the storage 
        protocol to specify how these are to be used.  There are ranges 
        for blocks-based storage protocols, for object-based storage 
        protocols and a reserved range for other experimental storage 
        protocols.  The RFC defining such a storage protocol needs to
        specify how particular bits within its range are to be used.  
        For example, it may specify a mapping between attributes of 
        the layout (read vs. write, size of area) and the bit to be 
        used or it may define a field in the layout where the associated
	  bit position is made available by the server to the client.

        When an undefined bit is set in the type mask, NFS4ERR_INVAL
        should be returned.  However even if a client does not support
        an object of the specified type, if the bit is defined, 
        NFS4ERR_INVAL should not be returned.  Future minor versions
        of NFSv4 may expand the set of valid type mask bits.

        CB_RECALL_ANY specifies a count of objects that the client may
        keep as opposed to a count that the client must return.  This
        is to avoid potential race between a CB_RECALL_ANY that had a 
        count of objects to free with a set of client-originated 
        operations to return layouts or delegations. As a result of the 
        race, the client and server would have differing ideas as to how

        many objects to return.  Hence the client could mistakenly free 
        too many.

        If resource demands prompt it, the server may send another
        CB_RECALL_ANY with a lower count, even it has not yet received
        an acknowledgement from the client for a previous CB_RECALL_ANY
        with the same type mask.  Although the possibility exists that
        these will be received by the client in a order different from
        the order in which they were sent, any such permutation of
        the callback stream is harmless.  It is the job of the client
        to bring down the size of the recallable object set in line
	  with each CB_RECALL_ANY received and until that obligation is
        met it cannot be canceled or modified by any subsequent 
        CB_RECALL_ANY for the same type mask.  Thus if the server 
        sends two CB_RECALL_ANY's, the effect will be the same as 
        if the lower count was sent, whatever the order of recall
        receipt.  Note that this means that a server may not cancel
        the effect of a CB_RECALL_ANY by sending another recall with
        a higher count.  When a CB_RECALL_ANY is received and the
        count is already within the limit set or is above a limit 
        that the client is working to get down to, that callback has no
        effect. 

        The client can choose to return any type of object specified 
        by the mask.  If a server wishes to limit use of objects of a
        specific type, it should only specify that type in the mask
        sent.  The client may not return requested objects and it is 
        up to the server to handle this situation, typically by doing
        specific recalls to properly limit resource usage.  The server 
        should give the client enough time to return objects before 
        proceeding to specific recalls. This time should not be less
        than the lease period.

        Servers are generally free not to give out recallable objects
        when insufficient resources are available.  Note that the 
        effect of such a policy is implicitly to give precedence to
        existing objects relative to requested ones, with the result
        that resources might not be optimally used.  To prevent this, 
        servers are well advised to make the point at which they start 
        issuing CB_RECALL_ANY callbacks somewhat below that at which
they
        cease to give out new delegations and layouts.  This allows the 
        client to purge its less-used objects whenever appropriate and
        so continue to have its subsequent requests given new resources
        freed up by object returns.

   ERRORS
        NFS4ERR_RESOURCE
        NFS4ERR_INVAL

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4