Subject: RE: [nfsv4] Eliminating stateid's from pnfs IO

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 6 Oct 2005 11:49:22 -0400

NFS allows directory caching.  NFS clients do it all the time.
There'd be no reason why an NFS client would *have* to do the
LOOKUP again.  And if it were the same principal doing the
open(), or a principal that had done the open() in the past
with legitimate cached attributes for the object, then
both of your wire operations might reasonably be skipped.

		Craig

> -----Original Message-----
> From: J. Bruce Fields [mailto:[email-masked]] 
> Sent: Thursday, October 06, 2005 11:30 AM
> To: Halevy, Benny
> Cc: Goodson, Garth; [email-masked]; Noveck, Dave
> Subject: Re: [nfsv4] Eliminating stateid's from pnfs IO
> 
> On Thu, Oct 06, 2005 at 03:23:46AM -0400, Halevy, Benny wrote:
> > J. Bruce Fields wrote:
> > > Absent delegations, I believe there *does* have to be a 
> NFS OPEN for 
> > > every open().  Since both open() and OPEN are a single atomic 
> > > operation including a lookup, there's no other way to ensure that 
> > > you get the right file.
> > 
> > Why? LOOKUP returns a filehandle uniquely identifying the 
> file based 
> > on atomic lookup.
> 
> Maybe it could be permissable to do just a LOOKUP and ACCESS 
> in place of the OPEN when not doing a create, if you've 
> already got an open with the identical share/deny bits.  I 
> don't think you can skip the LOOKUP, though, and if the 
> LOOKUP does reveal that the path no longer maps to the same 
> filehandle, then you're probably going to have to send an 
> OPEN anyway.  There doesn't seem to be much advantage to 
> allowing this.
> 
> --b.
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Thu, 6 Oct 2005 09:42:57 -0700 (PDT)

> NFS allows directory caching.  NFS clients do it all the time.

NFSv4 has no OPEN_DIRECTORY operation. Maybe it should. But it
doesn't, so comparing directories to regular files with regard to open
state has issues.

Anyway, I think you mean NFS allows name caching. Section 9.8
of RFC3530 discusses this.

> both of your wire operations might reasonably be skipped.

NFSv[234] clients can implement name caches for caching lookup
responses. There's no reason why they have to do an over the
network LOOKUP if they are opening non-regular files for NFSv4,
and all types of files for NFSv[23]. They do however have to
issue a GETATTR on the file being opened. At 
that point, they will learn if the file
objects contents have changes (via the change attribute), and also
if the file handle is valid (via NFS4ERR_STALE). What they don't
get to know is if the file has been renamed, which would have
made an over the network LOOKUP fail. Clients that care about this
need to issue a GETATTR on the parent directory to see if it has
changed. NFSv4, with its COMPOUND, gives v4 a decisive advantage in
that a COMPOUND that does a GETATTR on the parent directory and
the directory being opened can be done in one RPC. If the GETATTR on
the parent shows that the directory has changed, then this tells the
client it needs to re-validate the name cache entries for that parent
directory.

> > > > every open().  Since both open() and OPEN are a single atomic 

Yes, absent delegations this is the case: an OPEN for very open().

> > > > operation including a lookup, there's no other way to ensure that 

But that's not why there has to be an OPEN for every open(). Or
at least, that's not the primary reason for an OPEN for every open().

> > in place of the OPEN when not doing a create, if you've 

RFC3530:

"For a READ with a stateid value of all bits 0, the server MAY allow
   the READ to be serviced subject to mandatory file locks or the
   current share deny modes for the file.  For a READ with a stateid
   value of all bits 1, the server MAY allow READ operations to bypass
   locking checks at the server."

"   For a WRITE with a stateid value of all bits 0, the server MAY allow
   the WRITE to be serviced subject to mandatory file locks or the
   current share deny modes for the file.  For a WRITE with a stateid
   value of all bits 1, the server MUST NOT allow the WRITE operation to
   bypass locking checks at the server and are treated exactly the same
   as if a stateid of all bits 0 were used."

I.e. the server doesn't have to honor I/O that doesn't use a real stateid.

> > OPEN anyway.  There doesn't seem to be much advantage to 

If we had an OPEN that supported open by file handle (in steady state,
in addition to the OPEN by file handle that is supported in recovery
mode), then we'd have the path name revalidation issues we are
discussing. That said there would be an advantage to supporting 
OPEN by file handle. But I think I'd rather see directory delegations
in the next minor version before I'd like to see open by file handle.
If a client had a directory delegation, then open by file handle
has no consistency issues at all.

But this is really getting far afield of the nominal subject of
this thread: eliminating stateids from pnfs IO. Just say
no to eliminating stateids from any I/O, parallel or otherwise. You won't feel
dirty the morning after.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 6 Oct 2005 13:09:12 -0400

> I.e. the server doesn't have to honor I/O that doesn't use a real stateid.

But, for example, once you open a file for read, this doesn't prevent a
later open() from just using the existing open()'s state instead of
doing another open.

> no to eliminating stateids from any I/O, parallel or otherwise.

Why?

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 6 Oct 2005 13:38:30 -0400

> later open() from just using the existing open()'s state instead of

(I meant to write "another open for read" there.  Sorry about that.)

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Thu, 06 Oct 2005 13:58:20 -0400

[email-masked] said:
> But this is really getting far afield of the nominal subject of this thread:
> eliminating stateids from pnfs IO. Just say no to eliminating stateids from
> any I/O, parallel or otherwise. You won't feel dirty the morning after. 

fine. there are two ways to not elimiate stateid checking on IO for pNFS.

MDS = metadata server
DS = data server

1) [current design]
pNFS client OPENS to MDS. MDS constructs file layout consisiting of a list of  
<DSfilehandle,DSdevid> tuples. pNFS client presents MDSstateid in IO to each 
DS.

#1 implies a (new) backend protocol of managing the MDSstateid on each DS. you 
need to install/update/remove the MDSstateid.

2) [DSstateid in layout design] 
pNFS client OPENS to MDS. MDS constructs file layout consisting of list of 
<DSfilehandle, DSstateid, DSdevid> tuples. pNFS client presents DSstateid in 
IO to each DS.

#2 allows NFSv4.0 to be the backend storage protocol between MDS<=>DS. #2 also 
allows choice #1 functionality, simply use the MDSstateid as each DSstateid. 
#2  also allows the use of special stateids.

if we are to do stateid checking on pNFS DS IO, what is the protocol design 
reason to exclude stateids from the file layout?

-->Andy

---

From: Garth Goodson <[email-masked]>
Date: Thu, 06 Oct 2005 11:27:59 -0700

> reason to exclude stateids from the file layout?

One reason, is that layouts would have to be recalled whenever the 
stateid embedded within the layout changed (e.g., if OPEN_DOWNGRADEs 
were propagated to the DSs).  The other question is whether you would 
use delegation stateids from data servers, probably not.  You probably 
wouldn't want to enable delegations on the DSs (only at the MDS), in 
which case I/O to the DSs wouldn't be subject to the same checks as in 
vanilla V4 and could not result in deleg. recalls in the same manner.

Additionally, data servers would not be able to validate whether clients 
hold valid layouts.  As well, writing to the proper location within 
sparse striped layouts can't be enforced (not a terrible problem as you 
can use dense layouts).

In summary, I think there are a number of validation steps that the DSs 
are unable to make (above are a few), so it becomes a question of how 
far are you willing to go for what gains.

As well, would the unmodified data servers be using sessions?  If not 
that opens up another can of worms.  If so, then they could just as 
easily be updated to handle some of the added pNFS semantics.

-Garth

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Thu, 06 Oct 2005 14:29:39 -0400

> reason to exclude stateids from the file layout?

ok - trond reminded me. layout last over many opens.

-->Andy