Subject: guarded SETATTR atomicity in Linux

From: Neil Brown <[email-masked]>
Date: Thu, 21 Jun 2001 14:40:56 +1000 (EST)

Spencer asked me off-list about how the Linux NFS server can (once my
patch goes in) achieve atomicity of the SETATTR guard test with the
SETATTR of the size.  I thought others might be interested so I am
copying to the list.

The answer is that it come for free...

In Linux, each "struct inode" has a semaphore (i_sem) which is used to
serialise all changes to the file or directory.
So when writting to a file, or truncating it, the semaphore is held
while the change is made (at least while it is made to the in-memory
cache.  Changes might not get out to disc until well after the
semphore is released).  The ctime (and mtime) are updated while the
semaphore is held.

When calling "notify_change" - which passes metadata changes down to
the file system - the caller must have claimed i_sem if a change of 
ATTR_SIZE is requested.

So nfsd already claims the semaphore if ATTR_SIZE was set:

	if (iap->ia_valid & ATTR_SIZE) {
		fh_lock(fhp);
		size_change = 1;
	}

and releases it later:

	if (size_change) {
		fh_unlock(fhp);
		put_write_access(inode);
	}

We simply test the guardtime inside these two and avoid calling
notify_change if the time is wrong.

Unfortunately it is not possible using the same locking rules to
atomically get pre-op and post-op attributes for WRITE.
This is beause the ->write methods claim the i_sem when they need it,
so nfsd cannot claim it first.   This contrasts with the ->setattr
method called by notify_change, which does not claim it first.

NeilBrown

Patch is viewable at
  http://www.cse.unsw.edu.au/~neilb/patches/linux/2.4.6-pre4/patch-J-nfs3setattr

---

From: Som <[email-masked]>
Date: Thu, 21 Jun 2001 23:15:28 -0700 (PDT)

1.
In section 4.2.2 Persistent filehandle is defined as:

".... having a fixed value for the lifetime of the
file system object to which it refers"

Is this also true when the file system object is
removed? As per the specification probably that's not
true. But in the next chapter it says server must
return an error when a removed persistent file handle
is presented. Only way server can implement this is to
store even the deleted file system handle for its
lifetime....I mean, as per the implementation goes
server can not relocate a persistent filehandle even
after its deleted, since it has to return error in
future.

But as per the definition of persistent filehandle
goes, its only valid within the lifetime of filesystem
object!

I see a contradiction here...
To get rid of the contradiction one may modify section
 4.2.2 like...

".... having a fixed value for the lifetime of the
file system (note: not object ) to which the object
belongs"
But that's going to be very difficult to
implement...so probably a practical timeout value has
to be decided on persistent filehandles after
deletion!

2. Nothing is told about migration/replication of
deleted persistent filehandles..should that
information be also migrated/replicated? Probably yes,
but the didn't find that in spec..

thanks, som.

__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 22 Jun 2001 03:42:20 -0500

> goes, its only valid within the lifetime of filesystem

Right.  Later in that section it explicitly states:

  "The persistent filehandle will be become stale or invalid when the
   file system object is removed.  When the server is presented with a
   persistent filehandle that refers to a deleted object, it MUST return
   an error of NFS4ERR_STALE."

> to be decided on persistent filehandles after

I can't seem to find the reference in the RFC at the moment but there
are comments somewhere that talk about the eventual reuse of a
persistent filehandle and that the server should wait as long as
possible before such reuse.  If these statements are not in the
current RFC, I will make sure they are in the update.

In any event, this description takes into account the common Unix
implementation of NFS filehandles.  Historically, the filehandle
contains the inode number and generation number of that particular
inode.  It also contains other identifiers but these are the most
important.  If the inode is reallocated after the filesystem object is
removed, the generation number is incremented to signify a distinct
filesystem object.  That generation number will eventually wrap but it
has typically been a 32bit quantity so the time it would take to wrap
is "long".  With this mechanism, the server does store the necessary
persistent state to avoid duplicate filehandles for different objects
and that state is in the filesystem itself.

> but the didn't find that in spec..

Right.  It is implied in that if the filesystem is migrated and the
filehandles for that filesystem are presented as persistent, then the
new server will employ similar filehandle allocation techniques.
Maybe the RFC is a sparse in this area but I believe the current
intent is sufficient but maybe I am a little too close to the RFC as
it stands. :-)

-- 

- Spencer -

---

From: Som <[email-masked]>
Date: Sat, 23 Jun 2001 08:24:07 -0700 (PDT)

Thanks Spencer!

I must admit that your email was more clear than the
spec :-) ( probably calls for inclusion in spec ).

I have some more doubts:

1) Can a NFS client assume that a persistent
filehandle is never allocated ( after its deleted ) as
a volatile filehandle? 
[ In section 4.2.4 of the spec it tries to define the
format of volatile filehandle...but, was it part of
spec or just an example? so looks to me, a persitent
filehandle can never be equal to a volatile filehandle
..is that true? ]

2) Can a NFS server steal few bits of persistent
filehandle to denote as "reincarnation bits"?...the
purpose of this is to inform client that a "deleted
persistent filehandle" [ DPF ] being re-used ( so
don't trust this handle anymore...a new error status
is needed in that case ).

If we don't do 2), then a NFS client may misuse a DPF
handle thus corrupting a file.
[ however, reincarnation bits don't fix the problem
100%, but its better than no information at all ]

3) NFS clients need not know anything about those
special bits in persistent filehandle, but the new
error status will be needed.

thanks, som.

> intent is sufficient but maybe I am a little too

=====
"Chess holds its master in its own bonds,
shackling the mind and brain so that the 
inner freedom of the very strongest must 
suffer"-Albert Einstein