Subject: [nfsv4] pnfs update (march 05 connectathon meeting)

From: Brent Welch <[email-masked]>
Date: Fri, 04 Mar 2005 13:44:37 -0800

We had a small gathering at Connectathon to talk about pnfs.
The main highlight is that Garth Goodson has prototyped a
pnfs server inside a NetApp filer, and the good folks at Sun
demonstrated interoperability with that server from a hacked
Solaris 10 client.  Wow!  This implementation uses NFSv4 as
the storage protocol between the client and other files.  It
also uses NFSv4 as the "management" protocol between the pNFS
(i.e., metadata server) and the data servers.

The layout information for the file is simply an array of
filehandles and stateids, and a stripe size (e.g., 64K) and
file data is striped across the container files.  When a file is
created, the pNFS server creates the backing files for data on
the data servers, and a local empty file that just stores attributes.
Garth can expound on the details for those that are interested.

I attempted, and failed, to get draft-welch-pnfs-ops-01.txt
submitted before the IETF-62 deadline.  I believe it will appear
after the meeting next week in Minneapolis.  In the meantime,
Spencer suggested I (temporarily) self-publish, so you can find it
http://www.beedub.com/draft-welch-pnfs-ops-01.txt

The primary updates relative to -00.txt are in the framing of
the discussion about using other storage protocols, in the security
implications of other protocols, and the addition of a more detailed
description of the object storage security protocol.

During the Connectathon meeting we talked about where we wanted the
drafts to go next.  We came up with the following bullet points

* The simple NFSv4 layout (along the lines of Garth's prototype)
should be in the main pnfs-ops document as a REQUIRED layout.

* Separate drafts for block storage and object storage pnfs layouts.
Welch to own the object storage draft, David Black to own the
block storage draft. 

* More verbage on error recovery, which now has three parties that
can fail: the metadata server, the client, and the data server.

* More verbage on the interaction with locking.  We noted that for
advisory locking there is nothing to do.  For mandatory locking
the metadata server can recall layouts to disallow direct I/O
that conflicts with a lock. 

* More verbage on interaction with SetAttr.  E.g., a truncate may
require recall of a layout

* Goal to have all these done before IETF-63 (Paris)

In addition we debated these topics:

* What happens when LAYOUTCOMMIT fails?  What does that mean?
At this point the client has done the writes to storage, but the
metadata server is confused.  The best you can do is get a new
layout and re-apply the writes (or be clever if the layout is the same)

* Should the client hint to the metadata server at the time of
LAYOUTRETURN that they encountered errors using the layout.
Welch thinks this is important to avoid forcing the metadata
server to touch storage (e.g., do extra GetAttrs) in the normal path.

* How does having lots of clients (1000's) affect the grace period
upon system restart?  Will we need progressively larger and larger
grace periods?  This isn't a strictly pNFS issue.

* The block world has an "eof push" type of callback that is used
to update other clients about changes in the eof position of a file.
We think this is just an optimization/cache consistency feature that
isn't necessary for pNFS clients.  Because of block-oriented buffer
caches, it is easy to prefetch extra data into the buffer cache,
have a truncate done by another client (not even through pNFS) and
not "see" that truncate immediately.  Oh well, that's NFS semantics
and the use of pNFS doesn't change the picture.
--
Brent Welch
Software Architect, Panasas Inc
Delivering the premier storage system for scalable Linux clusters

www.panasas.com
[email-masked]

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Dean Hildebrand <[email-masked]>
Date: Fri, 18 Mar 2005 19:04:19 -0500 (EST)

Great work on the document Brent.

I have some comments regarding the wording.
Section 1, page 4:
  Generally, a single user Client's
  misbehavior can only impact files accessible to that single user.
  Misbehavior by a multi-user Client may impact files accessible
  to all of its users.  Delegations increase the level of Client
  responsibility as a client that carries out actions requiring a
  delegation without obtaining that delegation will cause its user(s)
  to see unexpected and/or incorrect behavior.
--
I find the use of upper case Client and lower case client here and in
other parts confusing without prior definition.  I can see different
possibilities for what multi-user client means, and I'm pretty sure I know
what you mean, but maybe it should also be more clear. (multiple threads
on a single machine, multiple machines, etc)

Section 1, page 5
To avoid any confusion between the existing
  NFSv4 data delegations and layout delegations, the term "layout"
  implies "layout delegation".
--
Why not always just say "layout delegation"?  If layout means layout
delegation, how do you talk about a layout (file, block, etc)?  Also, did
we resolve that it is impossible for a client to have a layout without a
delegation? (which I think this naming might imply)

Section 8.1:
The more I think about layoutget_iomode4 and layoutget_sharemode4, I'm
wondering why we aren't following the access/deny strategy of NFSv4/CIFS?
Since LayoutGet will most often come with an OPEN, it seems odd that
LayoutGet doesn't use the same system.  Maybe its just semantics, but
since the model is already in place...

I agree with the comments in the document that more discussion is needed
for the multiple shared readers and shared writers case.  Maybe it is just
an extension of the possible error codes.

Also, to be consistent with NFSv4, shouldn't
LAYOUTGET_RW
be renamed to
LAYOUTGET_BOTH

Dean Hildebrand
University of Michigan

---

From: Brent Welch <[email-masked]>
Date: Mon, 21 Mar 2005 09:29:01 -0800

Thanks!  looks like we need a careful definition section.

>>>Dean Hildebrand said:

www.panasas.com
[email-masked]