Subject: [nfsv4] IETF-65 pNFS notes

From: Brent Welch <[email-masked]>
Date: Mon, 27 Mar 2006 21:26:32 -0800

There was an informal pNFS meeting between nfsv4 sessions at IETF-65.
Here are my notes:

The first issue was the discussion of Commit handling in the files-based
layout.  This is being discussed on list currently.  The summary is that
for some files-based layouts it is perferred that COMMIT goes to the
metadata server (i.e., MDS, i.e., the file server) instead of having
COMMIT operations go to each data server.  The implication is that in
this case the MDS will communicate with the data servers to ensure the
COMMIT happens, and can arrange certain optimizations.  This alternative
will be flagged by a property of the file-based layout returned to the
client.  Note that this is distinct from LAYOUTCOMMIT.  This is about
clients doing async writes to the data servers, and then wanting the
back end storage system to COMMIT that data.  This all happens before
the LAYOUTCOMMIT.

Next we discussed the issues raised by truncating
set attribute operations.  The sequence that causes problems is when
a client accepts data into its buffer cache that grows the file,
then a truncating set attributes moves the EOF position before the
end of the dirty data.  And finally, sometime later writes are done
out beyond the end of file.  This can create a whole in the file that
must (MUST) be zero-filled.  The corner cases get worse with pNFS where
clients may be writing data that the metadata server isn't aware of, yet.
The simplest way to deal with this scenaario is to recall layouts so that
clients are not updating storage concurrently with the truncating setattr.
Battle scars from block- and object-based file systems lead to the
general advice that trying to "optimize" this scenario is a recipie
for trouble.

[As I write this, I realize we should have text that explains when the
CB_SIZECHANGED callback should be used, and when the layout should
simply be recalled with CB_LAYOUTRECALL.  The current ID says that
CB_SIZECHANGED is preferred to recalling the layout.  And
my notes say that the server should do CB_SIZECHANGED before
responding to a setattr request.  But my dire warnings above indicate
that a CB_LAYOUTRECALL is more robust.  This may be layout specific.]

We discussed the TODO list for the Linux client implementation.
The pNFS modifications were recently brought up-to-date with the
2.6.16 code stream (-rc6, to be precise).  The 2.6.16 nfsv4 is
available in the Fedora Core 5 release.  There are rumors that
SUSE may have a release based on 2.6.16 by June.
The biggest TODO item is CB_LAYOUTRECALL.  Followed closely
by Sessions support.  There is a sessions patch based on
the 2.6.9 code base, and we speculated that it should be "easy"
to apply it to 2.6.16.

Later during the IETF meeting there was a discussion about whether
sessions support should be "mandatory for 4.1", and in general what
minimum set of features ought to be required for an implementation
to claim "4.1-ness".  I'm not sure we bottomed out on that.
I will editorialize here and point out that sessions have a number
of nice properties:
  They help the server maintain the correctness in the form of
  a useful duplicate reply cache
  They help RDMA implementations that lead to improved performance
  They can help implement the necessary race detection associated
  with the callback protocols in NFSv4

Back at the pNFS meeting...

We noted that the security mechanisms for pNFS, as well as interactions
with data delegations, are also on the TODO list to ensure we gain
the necessary implementation experience.  That led us to a discussion
of the next interoperability testing.  It looks like that will be a
Bakeathon hosted by CITI at UMich in September.  A preliminary set
of things to test include (note a number of pure NFSv4.0 items):
  FS_LOCATIONS interoperability
  SECINFO, expecially the interaction with netgroups and exports in Linux
  ACLS ?
  pNFS (nothing more specific than this was suggested)
  Directory Delegations
  Sessions

Andy Adamson wants to try and arrange testing partners so folks can
bring implementations and have somebody else to test their particular
feature against.  We agreed that by the July IETF this kind of arrangement
will be more urgent, and folks will have a better idea of what they
will be able to test.
--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4