Subject: [nfsv4] LAYOUTCOMMIT discussion

From: <[email-masked]>
Date: Tue, 27 Jul 2010 15:30:06 -0400

I'd like to make sure we pursue this and come to some sort of conclusion
or at least make progress toward a solution.

One problem with LAYOUTCOMMIT description is that the text is very
block-oriented.  That's a result of the fact that 80% of the
functionality is appropriate to block layouts but it needs to explain,
what is and what isn't done in the case of file layouts.  A particular
example is loca_last_write_offset.  This is required because for block
because you may write one block of a file when only one byte is part of
the file.  But, for file, the DS knows the last offset and allowing you
to specify a possibly invalid one is not appropriate. 

But as far as the issue about when and if to use it, the troubling part
is the following last paragraph of section 13.10.

   The NFSv4.1 protocol only provides close-to-open file data cache
   semantics;

This is NOT what the v4.1 protocol provides.  It is up to the client to
determine if it caches data and many clients, don't for good reasons.
As far as the protocol goes, leaving pNFS aside, if you do a write the
change attribute will change and it supposed do. 

   meaning that when the file is closed, all modified data is
   written to the server.  When a subsequent OPEN of the file is done,
   the change attribute is inspected for a difference from a cached
   value for the change attribute.  For the case above, this means that
   a LAYOUTCOMMIT will be done at close (along with the data WRITEs) and
   will update the file's size and change attribute.  Access from
   another client after that point will result in the appropriate size
   being returned.

"a LAYOUTCOMMIT 'will be done' at close".  Not MUST or SHOULD.  And the
text suggests that the client adds this op as opposed to it being part
of CLOSE, but it isn't absolutely clear about that.

The reason that it is better for this to be automatic part of CLOSE, is
that there are going to be instances in which the file is closed when no
CLOSE is done (e.g. lease expiration), and the server had better do a
LAYOUTCOMMIT at that point or everything is a mess.

So here is my proposal for what should be written in this place:

    Many uses of the NFSv4.1 protocol are based on close-to-open
    caching semantics and it is a requirement that the modify time
    is updated frequently enough to support this.  In the case of
    the file layout type, when there have been writes done through
    a file layout on a file being closed, the client SHOULD do a 
    LAYOUTCOMMIT after completion of all the writes and before the
    CLOSE.  This will ensure that the change attribute and file size
    are updated appropriately so that access from another client
    after that point will result in the appropriate attributes
    being returned.

    When a file open for write is closed a result of a lease expiration,

    and the client has a layout for that file, the server must perform 
    the equivalent of a LAYOUTCOMMIT before closing the file, typically
    by fetching current attributes from the DS, to ensure that the
file's 
    attributes are properly updated.  When a lease expiration involves
    multiple files open for write for which there are associated
    layouts, the server is free to fetch attributes for multiple
    file from a DS in a single control protocol request. 
    
    When a lease is expired but the close does not occur immediately 
    the server still SHOULD perform the equivalent of a LAYOUTCOMMIT 
    at the point of lease expiration.

    In many applications, there are files that essentially are never
    closed.  In order to provide more meaningful attribute values in
    such environments, the server SHOULD ensure that the propagation
    of attributes (size, change, etc.) is done at least every lease
    period.  Because attribute updates for many file from a single DS 
    could be gathered in a single request, the resources to do this
    not very large.

 Comments?

---

From: "Everhart, Craig" <[email-masked]>
Date: Wed, 28 Jul 2010 02:25:49 -0400

Dave's great points about whether LAYOUTCOMMIT should be an implicit
part of implicit CLOSEs are useful, and I agree we ought to talk about
them.

As I understood at least one discussion on Monday around the lunch
table, the files-oriented version of pNFS felt that LAYOUTCOMMIT might
be optional if implemented on some kinds of file systems, in particular
many clustered back ends.  As I recall, the concern was whether simply
doing WRITEs and COMMITs to the file-oriented data servers would simply
sufficiently update the mtime and change attributes, as returned by the
MDS, or whether a separate LAYOUTCOMMIT was necessary to communicate
those changes back to the MDS.

If that's a reasonable capturing of the semantic issue involved, then it
may be straightforward to propose prose and semantics to cover it, such
as an option that the server can tell the files-oriented client saying
that LAYOUTCOMMITs are not required after committed writes to the data
servers in a given MDS/DS regime.

I look forward to discussions on these points, perhaps in person here in
Maastricht.

		Craig

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Tuesday, July 27, 2010 9:30 PM
> To: [email-masked]
> Subject: [nfsv4] LAYOUTCOMMIT discussion
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Wed, 28 Jul 2010 02:31:03 -0400

Good point.

There are certainly clustered file systems where none of this would be
required.

My intention was to specify a minimal set of occasions when the
attribute updates occurred.  Having it happen more frequently, or
always, as would be the case in a clustered file system, was certainly
intended to be allowed.  I guess there should be some text explaining
that.

-----Original Message-----
From: Everhart, Craig [mailto:[email-masked]] 
Sent: Wednesday, July 28, 2010 2:26 AM
To: Noveck, David; [email-masked]
Subject: RE: [nfsv4] LAYOUTCOMMIT discussion

		Craig

---

From: "Everhart, Craig" <[email-masked]>
Date: Wed, 28 Jul 2010 03:28:15 -0400

Not to belabor, but I was trying to amplify not only by saying that in
some cases the implicit LAYOUTCOMMIT isn't required, but that there are
server architectures in which LAYOUTCOMMIT is never required in ordinary
operation either, and that there's some call for having clients freed
from the need to do the MDS communication overhead in such cases.

		Craig

> Sent: Wednesday, July 28, 2010 8:31 AM
> To: Everhart, Craig; [email-masked]
> > that there are going to be instances in which the file is closed
when
> > no

---

From: <[email-masked]>
Date: Wed, 28 Jul 2010 05:16:22 -0400

OK.  Let's put it this way.  If we accept for the moment that I have
articulated all the cases in which LAYOUTCOMMIT activity needs to be
done for pNFS-file, they can be split into two classes:

1) Those that are incumbent on the server

2) The one that is before CLOSE

Given that, as Craig says, it makes sense to move the attribute update
to the server as part of close and not asking the client to do something
that may be a no-op on many servers.  Asking people to insert the
LAYOUTCOMMIT would add interoperability issues if people develop clients
and test with servers have clustered file systems.  In that environment
the lack of LAYOUTCOMMIT would not matter but it very well might when
running with a different server/FS.

The other issue with this is the level of requirement.  If this were a
MUST, then moving it from the client to the server should not be much of
an issue.  Right now I have it as SHOULD (although I can't say why) so
moving it from client to server could be an issue.

-----Original Message-----
From: Everhart, Craig [mailto:[email-masked]] 
Sent: Wednesday, July 28, 2010 3:28 AM
To: Noveck, David; [email-masked]
Subject: RE: [nfsv4] LAYOUTCOMMIT discussion

		Craig

when

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Wed, 28 Jul 2010 09:30:47 -0400 (EDT)

Hi,

Craig's initial comment I think suggested the notion of the server indicating its layoutcommit requirement(s).  Intuitively, this seems helpful.

Matt

----- "Noveck David" <[email-masked]> wrote:

> > If that's a reasonable capturing of the semantic issue involved,
> then
> > it
> > I look forward to discussions on these points, perhaps in person
> here
> > in
> > Maastricht.
> > 
> > 		Craig

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309