Subject: Re: [nfsv4] A more coherent data cache for NFS4.3 clients

From: "Matt W. Benjamin" <[email-masked]>
Date: Mon, 20 May 2013 15:12:58 -0400 (EDT)

Hi,

In combination with some type of data versioning, it is possible to get better results.  I believe this concept exists in DFS as well as, of course, in AFS.

In addition to the 3 cases Marc mentions, the default case in the (extended version of the) AFS caching model (called "extended callbacks") that I worked on, is a stronger version of case 2, in which any client is a potential writer.  I think you get the same semantics as in Marc's case 2, if the presence of readers forces write synchrony (direct mode, I guess) on the writers.

Data versioning becomes more complex in the presence of pNFS, but there are obvious ways to unify them.

Matt

----- "Trond Myklebust" <[email-masked]> wrote:

> On Mon, 2013-05-20 at 10:33 -0700, Marc Eshel wrote:
> > The objective of this feature is to improve NFS client caching for
> > NFS4.3 
> > I would like to find out from the group if one, others see this as
> a
> > useful 
> > addition to NFS, and two, if extending the delegations feature is
> the
> > right 
> > approach. 
> > 
> > The requirements as I see them is to make NFS client caching
> possible
> > while 
> > maintaining its coherency without changing the NFS client
> > applications. Have 
> > better synchronization of data among applications on different NFS
> > clients 
> > while caching data. 
> > 
> > The options that we have today for more synchronization of data
> among
> > NFS 
> > clients is direct IO which means no data caching. With delegations
> we
> > can have 
> > caching on multiple NFS client readers or a single NFS client
> writer. 
> >   
> > It would be nice to make better use of delegations and more
> > specifically the 
> > combination of read write delegation. In general the just read
> > delegations are 
> > useful but the combination with write delegation are less useful
> for
> > most cases. 
> > 
> > Selecting delegation as the synchronization method and not byte
> range
> > locks 
> > have few reasons. The first objective is to not block applications
> > from read or 
> > write but to just synchronize the updates, this extension need to
> > coexist with 
> > current delegation and should be considered as a more refined
> subset
> > of the 
> > current  delegations, and finally because we don't want to requiter
> > any changes 
> > to the application. 
> > 
> > One problem with delegation is that it is server and not client
> > initiated, so we 
> > might need to add some mount options or hints for the applications
> to
> > request 
> > this new behavior, or adding the option for the NFS client to
> request
> > a byte range 
> > delegation. 
> 
> Aside from the cases where O_DIRECT and locking are active, when
> would
> the client not want to request a delegation for something that it is
> reading?
> 
> > Delegation will be extended to include a byte range and delegation
> > callback 
> > will have to specify the part of the file that were changed by a
> > writer or being 
> > requested by  a reader. 
> > 
> > It might be required that the byte range be on block boundary. The
> > byte range 
> > might start with full file delegations, but it might be reduced to
> the
> > range that 
> > represent the cached portion of the file. We will have to put some
> > limit to the 
> > number of byte ranges to keep resources usage under control. 
> > 
> > The applications that can benefit from this feature are HPC
> > application, multi-node 
> > database programs, web server cluster, etc. 
> > 
> > 
> > case 1: 
> > Lets consider the case of multiple readers and one writer. In this
> > multi-reader-one-writer 
> > mode all clients will keep both the delegations and the cache and
> will
> > start with a full 
> > file byte range. 
> > When a reader makes a read of some byte range from the server the
> > server will make a 
> > callback to the writer, if it is holding a delegation for the
> > requested read byte range, 
> > requesting that this range be flushed to the server, the writer
> will
> >  flush the changed 
> > byte range if the range is dirty before the server can respond to
> the
> > reader. 
> > The writer might also consider give up that byte range to avoid
> more
> > call backs. 
> > 
> > case 2: 
> > Again all client in this multi-reader-one-writer mode will maintain
> > their caches. 
> > When the writer writes to the server all readers will be notified
> of
> > the changed byte range. 
> > At this point the reader can purge that range from its cache if it
> was
> > there in the first place 
> > or read that range from the server to keep its cache up to date. It
> > can also give up the 
> > delegation if it is not interest in caching this range. 
> > 
> > case 3: 
> > There are HPC application that split to file to segments that each
> > node is working on. 
> > It will be nice to give each client a write delegation for the
> portion
> > of the file that is will 
> > update, a good hint would be when the client punches a hole. 
> 
> ???? Why is that a hint?
> 
> Either way, please see
> 
> https://datatracker.ietf.org/doc/draft-myklebust-nfsv4-byte-range-delegations/
> 
> for a draft implementation of the subfile delegation concept.
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> [email-masked]
> www.netapp.com
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: "Myklebust, Trond" <[email-masked]>
Date: Mon, 20 May 2013 19:53:42 +0000

> Data versioning becomes more complex in the presence of pNFS, but there are obvious ways to unify them.

Please see the 'change_attr_type' attribute. If that is set to
NFS4_CHANGE_TYPE_IS_VERSION_COUNTER, then you effectively have data
versioning through the change attribute.

NetApp

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Mon, 20 May 2013 16:03:52 -0400 (EDT)

Yes.

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309