Subject: Re: [nfsv4] RFC 7530: Filehandle of opened file after the REMOVE

From: David Noveck <[email-masked]>
Date: Sun, 1 Jan 2017 08:13:55 -0500

> We could still persist that information
> somewhere, or use a flag to delay the deletion of unlinked inodes until
> NFSD runs.

You need to delay the deletion until after the grace period is over.  There
may be technical issues with this but the biggest hurdle may may be
architectural/political.  I expect that people involved in file system
recovery may have trouble with the idea that the NFS server is telling them
when they can and cannot do various pieces of file system recovery.

It might be more possible to ask them to complete recovery but move the
unlinked inodes to a directory like /unlinked-inodes and consider the
recovery complete before NFS actually runs.  If that is OK, and I believe
it is, the result would be almost the same as the NFS server moving the
file to that directory in the first place, using a server-based silly
rename.

> Personally I'd love to see sillyrename die.

I think you really want to see it dead.  Actually watching it die is
boring, sort of like watching paint dry

> It's a major pain for getting sensible semantics out of NFS.

Please elaborate.  Would a server-based silly rename allow the sensible
semantics you are looking for?

The best way to get silly rename to die is to provide an alternative, and I
believe an extension to NFSv4.2 could be defined to provide that.

> The unlinked inode list is almost a directory, except that it doesn't
> have names for the entries, you can only find inodes on it by the inode
> number and generation (aka NFS file handle).

The existence of those names could be considered a flaw, but if we define
an extension to allow silly rename to be eliminated, we should not REQUIRE
or RECOMMEND they they not be visible.  I'm thinking of some non-RFC2119
characterization such as "It is most desirable that these objects not be
accessible other than though  use of their file handles".

Btw, my preferred title for such a document, almost certain to be rejected,
is "Silly Rename Must Die" :-)

On Thu, Dec 29, 2016 at 2:48 AM, Christoph Hellwig <[email-masked]> wrote:

> On Wed, Dec 28, 2016 at 09:47:03PM -0500, Bruce James Fields wrote:
> > I never seriously worked on it, but for a while I was in the habit of
> > running it by people.  Christoph Hellwig thought it was doable (I think
> > he suggested some sort of callback from the filesystem during the
> > garbage collection, possibly because he had in mind some other
> > application for that--but my memory may be wrong).  Chris Mason didn't
> > like the idea at all.  He asked what we expect to happen on fsck, or if
> > the filesystem gets mounted without nfs getting started, or... some
> > other scenarios I forget.
>
> The way open but unlinked files are handled by modern transaction
> file systems is that the file system has a list of those inodes
> (in XFS this is the unlinked inode list in the allocation group header,
> other file systems use different terminologies and slightly different
> technics, e.g. in ext4 the list is global for the whole file system).
>
> After an unclean shutdown when file system recovery is run we'll perform
> the deferred delete for all the inodes on the unlinked inode list.
> At that point the file system could in theory inform NFSD about that
> fact.  But at least as far as the current Linux kernel is concerned (
> sorry for delving into implementation details, but I guess this is still
> easier to understand than an abstract discussion) at the point where
> file system performs recovery NFSD has not been started, or at least
> doesn't
> know about the file system yet.   We could still persist that information
> somewhere, or use a flag to delay the deletion of unlinked inodes until
> NFSD runs.
>
> > We could do the same silly rename tricks on the server side.  Something
> > like: create a directory with an unlikely name in the root of the
> > export, rename files there on REMOVE.  Possible problems:
>
> Personally I'd love to see sillyrename die.  It's a major pain for
> getting sensible semantics out of NFS.
>
> >       - you'll never be able to completely hide that directory.  But
> >         maybe we could get some sort of filesystem support for a
> >         hidden directory.
>
>
> The unlinked inode list is almost a directory, except that it doesn't
> have names for the entries, you can only find inodes on it by the inode
> number and generation (aka NFS file handle).
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Christoph Hellwig <[email-masked]>
Date: Sun, 1 Jan 2017 14:48:46 +0100

> when they can and cannot do various pieces of file system recovery.

Wearing my local fs hat the only issue I have is that I'd like to avoid
the case where people accidentally configure their fs to never reclaim
the unlinked but open inodes after a crash.

> file to that directory in the first place, using a server-based silly

For the classis text book case of file systems with intent log this
doesn't make sense.  Open but unlinked inodes are only tracked in the
unlinked inode list, but do not have a name attached to them any more.
So we'd have to do a lot of effort to move it into a directory using
synthetic names while we already have a better data structure built
for exactly this use case.  We'll just need to delay the action performed
on it a little bit.  And that actions is really trivial - basically the
inodes is read into memory and then released again to execute the same
code we'd execute after dropping the last reference to an open but
unlinked inode.

> semantics you are looking for?

After delving so much into internals above I'll switch to my protocol
developers hereand say: we should only specify the on the wire semantics,
how it's implemented on the server is not our business.

A NFS server that moves all files that are unlinked but open by a client
(note that this might be a problem for local opens or unlinks not going
through the nfs server) could provide semantics very close to the real
thing, but it would have to be creative about adjusting the link counts
sent to the client.

With my experience in local file systems and nfs servers I'd say it's much
simpler to let the file system do the hard work and let the NFS server
delay reclaiming of unlinked inodes after a crash in some way.

---

From: Tom Talpey <[email-masked]>
Date: Sun, 1 Jan 2017 17:32:04 -0500

On 1/1/2017 8:13 AM, David Noveck wrote:
> Btw, my preferred title for such a document, almost certain to be
> rejected, is "Silly Rename Must Die" :-)

Oh I don't know about that, look at the history for RFC6649.

https://datatracker.ietf.org/doc/rfc6649/

Type		RFC - Best Current Practice (July 2012; No errata)
		Obsoletes RFC 1510
		Updates RFC 4757, RFC 1964, RFC 4121, RFC 4120
		Also known as BCP 179
		Was draft-ietf-krb-wg-des-die-die-die (krb-wg WG)
Last updated	2015-10-14
Replaces	draft-lha-des-die-die-die
Stream		IETF

:-)

---

From: Rick Macklem <[email-masked]>
Date: Sun, 1 Jan 2017 23:19:28 +0000

Tom Talpey wrote:
> rejected, is "Silly Rename Must Die" :-)
Actually replying to a random thread, mostly Dave's...

A case that I wasn't clear on w.r.t. this is:
- The client holds a delegation for the file.
- The client issues opens locally w.r.t. this delegation.
- A process/thread on the client does a "Remove" of the file.

Is the client expected to acquire Opens against the server and return the
Delegation? (This seems like more work than just doing good old sillyrename?)
- Or does a Delegation have the same effect as an Open against the file on the
  server.

rick

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 2 Jan 2017 00:36:10 +0100

> - Or does a Delegation have the same effect as an Open against the file on

If you read RFC7530, then the client is expected to return the delegation
before doing any form of RENAME or REMOVE, since the server has no stateid
or session id with which to identify the origin of the RPC call.

It is less obvious that this needs be the case for NFSv4.x (x>0), but I
can't remember seeing any changes to those texts in RFC5661.

Cheers
  Trond

---

From: David Noveck <[email-masked]>
Date: Mon, 2 Jan 2017 06:54:36 -0500

> Actually replying to a random thread, mostly Dave's...

The big element of randomness/confusion is whether the questions with
regard to delegations are posed with regard to NFSv4.0, in which the client
is doing silly rename, NFSv4.1 in which OPEN4_RESULT_PRESERVE_UNLINKED is
returned, or to a hypothetical extension in which silly
rename is disallowed/deprecated.

As a result, the handling of your scenario is going to have a big dose of
"it depends" but I'll try to clarify as well as I can.

> Delegation? (This seems like more work than just doing good old
sillyrename?)

In the case of v4.0, no.  However, if the client is doing silly rename,
then the RENAME may trigger a delegation recall.  A client could choose to
return the delegation in advance to forstall the need for a recall and
reduce latency.  After all, if you are removing a file, the consequent file
non-existence assures you that no other clients have it open and the
delegation might well be considered superfluous. YMMV.

In the case of OPEN4_RESULT_PRESERVE_UNLINKED, the situation is unclear.
The existing text is not clear as to whether the server's promise to retain
the inode as long as the file is open
applies in the case in which a delegation allows a client-local open to
exist.  If we decide to try to revitalize
OPEN4_RESULT_PRESERVE_UNLINKED (unlikely in my opinion), it needs to be
clear that the potential
existence of local open needs to be taken into account.  However, if this
is done it should be clear that there
is no requirement to promptly resolve the question of the existence of such
local opens.

With regard to a hypothetical replacement for OPEN4_RESULT_PRESERVE_UNLINKED,
the handling
should be the same.

> Or does a Delegation have the same effect as an Open against the file on
the server.

That's the way things should be, if the server is making the decision, but
but we are not there yet.

---

From: David Noveck <[email-masked]>
Date: Mon, 2 Jan 2017 07:11:25 -0500

> If you read RFC7530, then the client is expected to return the delegation
before doing any form of RENAME or REMOVE

Could you be more specific about where you saw that?  I tried to find it
but didn't succeed.  In fact, with regard to RENAME, the text makes the
need for a recall conditional about whether the server in question might
disallow RENAMEs depending on what existing opens deny.  I don't remember
any statement about expectations regarding delegation return in advance of
these ops.

> since the server has no stateid or session id with which to identify the
origin of the RPC call.

True statement about an annoying aspect of v4.0, but I'm not clear about
where it appears in RFC7530.

Once we find the text in RFC7530, we can address RFC5661.  If the text you
recall seeing in RFC7530 appears unchanged in RFC5661, we might need an
errata.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 2 Jan 2017 13:52:48 +0100

> On Jan 2, 2017, at 13:11, David Noveck <[email-masked]> wrote:
> 
> > If you read RFC7530, then the client is expected to return the delegation before doing any form of RENAME or REMOVE
> 
> Could you be more specific about where you saw that?  I tried to find it but didn't succeed.  In fact, with regard to RENAME, the text makes the need for a recall conditional about whether the server in question might disallow RENAMEs depending on what existing opens deny.  I don't remember any statement about expectations regarding delegation return in advance of these ops.

See:
https://tools.ietf.org/html/rfc7530#section-10.4.4 <https://tools.ietf.org/html/rfc7530#section-10.4.4>

10.4.4 <https://tools.ietf.org/html/rfc7530#section-10.4.4>.  Recall of Open Delegation

   The following events necessitate the recall of an open delegation:

   o  Potentially conflicting OPEN request (or READ/WRITE done with
      "special" stateid)

   o  SETATTR issued by another client

   o  REMOVE request for the file

   o  RENAME request for the file as either source or target of the
      RENAME

   Whether a RENAME of a directory in the path leading to the file
   results in the recall of an open delegation depends on the semantics
   of the server file system.  If that file system denies such RENAMEs
   when a file is open, the recall must be performed to determine
   whether the file in question is, in fact, open.

Note that the above special case appears only to apply to _directories_. It is not qualifying the “RENAME request for the file as either source or target…”.

There is also https://tools.ietf.org/html/rfc7530#section-10.4 <https://tools.ietf.org/html/rfc7530#section-10.4>

   When a single client holds an OPEN_DELEGATE_READ delegation, it is
   assured that no other client may modify the contents or attributes of
   the file.  If more than one client holds an OPEN_DELEGATE_READ
   delegation, then the contents and attributes of that file are not
   allowed to change.  When a client has an OPEN_DELEGATE_WRITE
   delegation, it may modify the file data since no other client will be
   accessing the file's data.  The client holding an OPEN_DELEGATE_WRITE
   delegation may only affect file attributes that are intimately
   connected with the file data: size, time_modify, and change.

That last sentence implies (but does not state explicitly) that the client must give up the write delegation if it wants to modify other file attributes.

> 
> > since the server has no stateid or session id with which to identify the origin of the RPC call.
> 
> True statement about an annoying aspect of v4.0, but I'm not clear about where it appears in RFC7530.

It doesn’t, but that was the main argument for recalling the delegation in the cases.

> 
> > It is less obvious that this needs be the case for NFSv4.x (x>0), but I can't remember seeing any changes to those texts in RFC5661.
> 
> Once we find the text in RFC7530, we can address RFC5661.  If the text you recall seeing in RFC7530 appears unchanged in RFC5661, we might need an errata.
> 

https://tools.ietf.org/html/rfc5661#section-10.4.4 <https://tools.ietf.org/html/rfc5661#section-10.4.4>

The text in RFC7530 was a straight cut n’ paste job from RFC5661.

> 
> On Sun, Jan 1, 2017 at 6:36 PM, Trond Myklebust <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> On 2 January 2017 at 00:19, Rick Macklem <[email-masked] <mailto:[email-masked]>> wrote:
> Tom Talpey wrote:
> If you read RFC7530, then the client is expected to return the delegation before doing any form of RENAME or REMOVE, since the server has no stateid or session id with which to identify the origin of the RPC call.
> 
> It is less obvious that this needs be the case for NFSv4.x (x>0), but I can't remember seeing any changes to those texts in RFC5661.
> 
> Cheers
>   Trond
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
> 
>

---

From: David Noveck <[email-masked]>
Date: Mon, 2 Jan 2017 09:26:21 -0500

> See:
> https://tools.ietf.org/html/rfc7530#section-10.4.4

Thanks.

> Note that the above special case appears only to apply to _directories_.
It is not qualifying the “RENAME request for the file as either source or
target…”.

I got confused about this.  I think I somehow conflated this with the
material in the bulleted items in Section 16.26.5.  I somehow assumed that
the text would indicate that a delegation allowing local opens would have
to be recalled to enable the server to determine whether the directory
"SHOULD NOT" or "SHOULD" be deleted.  THere is no such  text in RFC7530
now, but I'm not sure what to do about that.

>There is also https://tools.ietf.org/html/rfc7530#section-10.4

> That last sentence implies (but does not state explicitly) that the
client must give up the write delegation
> if it wants to modify other file attributes.

I don't think that was the intention, although I can see how it might be
read that way.

I read this as saying that the client holding a WRITE delegation may only
affect those data-related attributes locally since it has a guarantee that
no other clients are looking at them (and that CB_GETATTR will be used when
appropriate).  Other attributes (including named attributes which are not
explicitly included or excluded) are outside the scope of the delegation.
I don't see that a client holding a WRITE delegation has to give it back
simple because he is changing some other attribute.  I think he can do the
SETATTR normally in this case.  I also note that section 10.4.4. says "SETATTR
issued by another client", suggesting that one issued by the client holding
the delegation is OK.

> https://tools.ietf.org/html/rfc5661#section-10.4.4

It is Section 10.4.4 in both 7530 and 5661.  Coincidence ... or
something else? :-)

>The text in RFC7530 was a straight cut n’ paste job from RFC5661

It appears that RFC5661 was a cut n' paste from RFC3530, so it isn't
clear whether the RFC7530 text was derived from RFC3530 (which makes
sense)  or from RFC5661, which doesn't, since v4.0 and v4.1 are different
protocols.

The problem is the inappropriate cut n' paste from RFC3530 to RFC5661.
It appears that this was a mistake although I don't think we can deal with
this
via an errata.  It is worth noting that the list in question is introduced
by the
the non-RFC2119 phrase "The following events necessitate the recall
of an open delegation".

On Mon, Jan 2, 2017 at 7:52 AM, Trond Myklebust <[email-masked]> wrote:

> > If you read RFC7530, then the client is expected to return the
> delegation before doing any form of RENAME or REMOVE
>
> https://tools.ietf.org/html/rfc7530#section-10.4.4
>
> Note that the above special case appears only to apply to _directories_.
> It is not qualifying the “RENAME request for the file as either source or
> target…”.
>
> There is also https://tools.ietf.org/html/rfc7530#section-10.4
>
> That last sentence implies (but does not state explicitly) that the client
> must give up the write delegation if it wants to modify other file
> attributes.
>
>
> https://tools.ietf.org/html/rfc5661#section-10.4.4
>
>

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 2 Jan 2017 15:42:28 +0100

> On Jan 2, 2017, at 15:26, David Noveck <[email-masked]> wrote:
> 
> > See:
> I got confused about this.  I think I somehow conflated this with the material in the bulleted items in Section 16.26.5.  I somehow assumed that the text would indicate that a delegation allowing local opens would have to be recalled to enable the server to determine whether the directory "SHOULD NOT" or "SHOULD" be deleted.  THere is no such  text in RFC7530 now, but I'm not sure what to do about that.
> 
> > That last sentence implies (but does not state explicitly) that the client must give up the write delegation 
> I don't think that was the intention, although I can see how it might be read that way.
> 
> I read this as saying that the client holding a WRITE delegation may only affect those data-related attributes locally since it has a guarantee that no other clients are looking at them (and that CB_GETATTR will be used when appropriate).  Other attributes (including named attributes which are not explicitly included or excluded) are outside the scope of the delegation.  I don't see that a client holding a WRITE delegation has to give it back simple because he is changing some other attribute.  I think he can do the SETATTR normally in this case.  I also note that section 10.4.4. says "SETATTR issued by another client", suggesting that one issued by the client holding the delegation is OK.
> 
> 
> It appears that this was a mistake although I don't think we can deal with this 
> via an errata.  It is worth noting that the list in question is introduced by the 
> of an open delegation”.

I fully agree that looks at first glance as if it is a non-normative text. Unfortunately, 15 years of server implementations have taken it as gospel, and hence no client that cares about performance can afford to ignore it.

> 
> On Mon, Jan 2, 2017 at 7:52 AM, Trond Myklebust <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> On Jan 2, 2017, at 13:11, David Noveck <[email-masked] <mailto:[email-masked]>> wrote:
>>