Subject: Re: [nfsv4] pnfs update (march 05 connectathon meeting)

From: Sujay Godbole <[email-masked]>
Date: Tue, 19 Apr 2005 21:58:27 +0530

Some comments on draft-welch-pnfs-ops-01.txt:

I have separated out into major comments and typos.
The following are major comments:

1) Section 8.1 LAYOUTGET:
    - 	"A shared layout allows cooperating clients to perform 
        direct I/O using a layout that potentially conflicts with 
        other clients. The clients are asserting that they are 
        aware of this issue and can coordinate via an external 
        mechanism (either NFSv4 advisory locks or, e.g., MPI-IO 
        toolkit)"

	[Comment]: what are the scenarios where this is useful and 
                   purpose behind this?

   -    For specifying additional ioctl operation to request shared 
        layout, imply existing application can not use this 
        additional facility.

2) Section 8.2 LAYOUTCOMMIT (Description part)
   -    While changing access and modify times using SETATTR whose 
        clock should be referred? (pNFS client or pNFS server).
        This imply need for clock synchronization 

3) Section 9.1 CB_LAYOUTRECALL.
   -    What are the scenarios of recalling the layout delegation?

   -   " The client should complete any in-flight I/O operations 
        using the recalled layout before returning it via LAYOUTRETURN.  
        If the client has buffered dirty data, it may chose to write it 
        directly to storage before calling LAYOUTRETURN, or to write it 
        later using normal NFSv4 WRITE operations."

        [Comments] : 

        - Does pNFS server support read & write requests from 
          standard nfsv4 clients. In that case server needs to 
          interpret layout and propagate read & write request to 
          appropriate storage servers. This also affects layout 
          delegation. 

        - So does this also imply that, if pNFS client failed to get 
          layout, he falls back to normal NFSv4 read & write 
          requests. This puts extra overhead on server and network, as 
          server has to fetch data to-and-fro from storage server and 
          give it back to client. Hence server is getting bottlenecked.
        
4) Layout Delegation:

   -    Since layout delegation is given on a byte-range in a 
        data file. This introduces new concept called byte 
        range data delegation. 

   -    Needs to explain various scenarios describing working 
        of Layout delegation with respect to data delegation, 
        which is already present in NFSv4. 

   -    Working of layout delegation with share reservation 
        should be mentioned

   -    More detail information required regarding how layout 
        delegation will work along with NFSv4 locking. 

   -    Can it possible if 'n' clients having layout delegation 
        on 'n' different files, and each client is appending to file. 
        Then comes new client and enquires about file system info, 
        how should pNFS server behave in order to get correct 
        free space count for the system?

   -    Suggested Conflict table for layout delegation & data delegation 

        ==================================================================================
        Client1:		Client2 :			Comment (Suggested behavior)
        Data delegation type	Layout delegation type	
        ==================================================================================
        Read 			Shared Read			layout delegation Allowed 
        Read 			Shared write			layout delegation Denied
        Read 			Shared both			layout delegation Denied
        Write			Exclusive read			layout delegation Denied
        Write			Exclusive write			layout delegation Denied
        Write			Exclusive both			layout delegation Denied
        ==================================================================================

   -    Similarly conflict table for layout delegation should 
        also be given.

5) 11.5 Nested Map
   It would be great if the data structures for Nested map can be 
   given.
    
6) Layout and aggregation:
   details information on this should be given regarding:
       
   -    who is responsible for taking decision regarding 
        aggregation.
   -    How to get aggregation information from server

----------------------------------------------------------------------

There are few minor formatting and typographical mistakes in the draft. 
They are listed below:

1) Section: 3 - Layouts and Aggregation
   "Even a one-to-one mapping use useful for 
[Comment]                   ( ^- possibly a typo.) 
   a file server that wishes to distribute its load among a set of  
   other file servers. There are also experimental aggregation types 
   such as writeable mirrors and RAID, however these are outside the 
   scope of this document."

2) Section 4.2 - Object Storage Security

   "Each capability is specific to a particular object, an operation 
   on that object, a byte range w/in the object, and has an explicit
[Comment]                      (^^^^ <- Is this within ?)
   expiration time.  The capabilities are signed with a secret key
   that is shared by the object storage devices (OSD) and the 
   metadata managers.  Clients do not have device keys so they are 
   unable to forge capabilities."

3) Section : 5.2 pnfs_deviceid4
   unsigned uint32_t pnfs_deviceid4;   /* 32-bit device ID */ 
[Comment] ^- could be just uint32_t instead of unsigned uint32 

4) Section 5.4 :
        struct pnfs_devlist_item4 {
        	pnfs_deviceid4   id;
        	nfs_deviceaddr4  addr;   
        /* [Comment]  Need to give definition of this. */
        };
       
5) Section 8.2 LAYOUTCOMMIT 

   "The client may use a SETATTR operation in a compound right after 
   LAYOUTCOMMIT in order to set the access and modify times of the 
   file.  Alternatively, the server could use the time of the 
   LAYOUTCOMMIT operation as the file modify time."  

[Comment]: Slightly confusing sentence. It would be good if word 'RPC' 
           is added after word 'compound'

Regards
Sujay Godbole 
Senior Engg
Calsoft pvt Ltd.

www.calsoftinc.com
[email-masked]

> Subject: [nfsv4] pnfs update (march 05 connectathon meeting)
> Date: Saturday 05 Mar 2005 3:14 am
> From: Brent Welch <[email-masked]>
> To: [email-masked]
> 
> We had a small gathering at Connectathon to talk about pnfs.
> The main highlight is that Garth Goodson has prototyped a
> pnfs server inside a NetApp filer, and the good folks at Sun
> demonstrated interoperability with that server from a hacked
> Solaris 10 client.  Wow!  This implementation uses NFSv4 as
> the storage protocol between the client and other files.  It
> also uses NFSv4 as the "management" protocol between the pNFS
> (i.e., metadata server) and the data servers.
> 
> The layout information for the file is simply an array of
> filehandles and stateids, and a stripe size (e.g., 64K) and
> file data is striped across the container files.  When a file is
> created, the pNFS server creates the backing files for data on
> the data servers, and a local empty file that just stores attributes.
> Garth can expound on the details for those that are interested.
> 
> I attempted, and failed, to get draft-welch-pnfs-ops-01.txt
> submitted before the IETF-62 deadline.  I believe it will appear
> after the meeting next week in Minneapolis.  In the meantime,
> Spencer suggested I (temporarily) self-publish, so you can find it
> http://www.beedub.com/draft-welch-pnfs-ops-01.txt
> 
> The primary updates relative to -00.txt are in the framing of
> the discussion about using other storage protocols, in the security
> implications of other protocols, and the addition of a more detailed
> description of the object storage security protocol.
> 
> During the Connectathon meeting we talked about where we wanted the
> drafts to go next.  We came up with the following bullet points
> 
> * The simple NFSv4 layout (along the lines of Garth's prototype)
> should be in the main pnfs-ops document as a REQUIRED layout.
> 
> * Separate drafts for block storage and object storage pnfs layouts.
> Welch to own the object storage draft, David Black to own the
> block storage draft.
> 
> * More verbage on error recovery, which now has three parties that
> can fail: the metadata server, the client, and the data server.
> 
> * More verbage on the interaction with locking.  We noted that for
> advisory locking there is nothing to do.  For mandatory locking
> the metadata server can recall layouts to disallow direct I/O
> that conflicts with a lock.
> 
> * More verbage on interaction with SetAttr.  E.g., a truncate may
> require recall of a layout
> 
> * Goal to have all these done before IETF-63 (Paris)
> 
> In addition we debated these topics:
> 
> * What happens when LAYOUTCOMMIT fails?  What does that mean?
> At this point the client has done the writes to storage, but the
> metadata server is confused.  The best you can do is get a new
> layout and re-apply the writes (or be clever if the layout is the same)
> 
> * Should the client hint to the metadata server at the time of
> LAYOUTRETURN that they encountered errors using the layout.
> Welch thinks this is important to avoid forcing the metadata
> server to touch storage (e.g., do extra GetAttrs) in the normal path.
> 
> * How does having lots of clients (1000's) affect the grace period
> upon system restart?  Will we need progressively larger and larger
> grace periods?  This isn't a strictly pNFS issue.
> 
> * The block world has an "eof push" type of callback that is used
> to update other clients about changes in the eof position of a file.
> We think this is just an optimization/cache consistency feature that
> isn't necessary for pNFS clients.  Because of block-oriented buffer
> caches, it is easy to prefetch extra data into the buffer cache,
> have a truncate done by another client (not even through pNFS) and
> not "see" that truncate immediately.  Oh well, that's NFS semantics
> and the use of pNFS doesn't change the picture.
> --
> Brent Welch
> Software Architect, Panasas Inc
> Delivering the premier storage system for scalable Linux clusters
> 
> www.panasas.com
> [email-masked]
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> -------------------------------------------------------
> 
>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4