Subject: Re: [nfsv4] [PATCH 2/2] nfs41: handle BLK_LAYOUT CB_RECALL_ANY

From: Benny Halevy <[email-masked]>
Date: Mon, 31 Oct 2011 19:08:43 +0200

On 2011-10-31 18:45, Trond Myklebust wrote:
> On Tue, 2011-11-01 at 00:38 +0800, Peng Tao wrote: 
>> On Mon, Oct 31, 2011 at 11:49 PM, Trond Myklebust
>> <[email-masked]> wrote:
>>> On Mon, 2011-10-31 at 08:15 -0700, Peng Tao wrote:
>>>> For blocklayout, we need to issue layoutreturn to return layouts when
>>>> handling CB_RECALL_ANY.
>>>
>>> Why?
>> Because replying NFS4_OK to CB_RECALL_ANY indicates that client knows
>> that server wants client to return layout. And server will be waiting
>> for layoutreturn in such case.
> 
> No it doesn't. NFS4_OK means that the client acknowledges that it has
> been given a new limit on the number of recallable objects it can keep.
> There is no requirement in the text that it should send layoutreturn or
> that the server should expect that.

The motivation for CB_RECALL_ANY is to reduce the state on the *server* side.
Quoting from RFC5661:
   The server may decide that it cannot hold all of the state for
   recallable objects, such as delegations and layouts, without running
   out of resources.  In such a case, while not optimal, the server is
   free to recall individual objects to reduce the load.
...
   In order to implement an effective reclaim scheme for such objects,
   the server's knowledge of available resources must be used to
   determine when objects must be recalled with the clients selecting
   the actual objects to be returned.
                      ^^^^^^^^^^^^^^
...
   When a given resource pool is over-utilized, the server can send a
   CB_RECALL_ANY to clients holding recallable objects of the types
   involved, allowing it to keep a certain number of such objects and
   return any excess.
   ^^^^^^^^^^^^^^^^^
...
   RCA4_TYPE_MASK_FILE_LAYOUT

      The client is to return layouts of type LAYOUT4_NFSV4_1_FILES.
                    ^^^^^^^^^^^^^^^^^

Isn't that explicit enough?

> 
> In any case, there is no reason to make a difference between block,
> object and file layouts when it comes to CB_RECALL_ANY. The code to
> handle it should be the same for all.
> 

Agreed.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 31 Oct 2011 13:42:44 -0400

>    the actual objects to be returned.

Leaving aside the fact that the above quotes contain no normative
language:
Right now, we do a bulk return of all layouts. Doing a layoutreturn for
each and every layout in that case is just ridiculous. Either do a
LAYOUTRETURN4_ALL after freeing all the layouts, or don't do anything at
all and just wait for the server to revoke the layouts for us (which is
what we currently do).
Both options should be faster than doing a LAYOUTRETURN4_FILE on each
and every file that is currently in use.

Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Benny Halevy <[email-masked]>
Date: Mon, 31 Oct 2011 19:57:07 +0200

> each and every layout in that case is just ridiculous. Either do a

The idea is to return the layouts for files that are the least used,
not each and every layout.

> and every file that is currently in use.

Doing LAYOUTRETURN4_ALL might cause a bug hiccup if the client needs to then send
a LAYOUTGET for each and every file that *is* currently in use.
So serving a CB_RECALL_ANY keeping more than 50% of the recallable objects means
the client would be better off returning the excess rather than returning everything
and reclaiming > 50% back.

Waiting for revocation may work well with some servers but would be disastrous in
terms of performance and responsiveness with others.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 31 Oct 2011 14:20:12 -0400

> terms of performance and responsiveness with others.

I don't necessarily disagree with what you are saying, but I have yet to
see a single server side implementation of CB_RECALL_ANY, let alone any
numbers that indicate performance or responsiveness problems resulting
from our existing client-side implementation.

I therefore find it hard to understand why optimising this particular
code is such a high priority, or why a patch that is adding per-file
layoutreturns to initiate_bulk_draining() is going to help anything at
all.

   Trond

NetApp

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Mon, 31 Oct 2011 14:23:10 -0400 (EDT)

Hi,

Such a server implementation will certainly not be long in coming.

Matt

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 31 Oct 2011 14:31:24 -0400

> Such a server implementation will certainly not be long in coming.

Good. However until then, we have nothing to test any performance claims
against.

Trond

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 31 Oct 2011 14:39:59 -0400

On Mon, 2011-10-31 at 14:31 -0400, Jim Rees wrote: 
> Trond Myklebust wrote:
> Testing between the linux block layout client and the EMC block layout
> server revealed a deadlock when the server had handed out some number of
> layouts and couldn't hand out any more.  So now the EMC block layout server
> implements CB_RECALL_ANY.  So yes, this solves a real world problem, and
> yes, there is a server that implements this.
> 
> We had some discussions at the time, and I don't remember if those were on
> the linux-nfs list or in some other forum.  We decided that the client was
> in the best position to decide which layouts were no longer needed, so we
> needed some way for the server to tell the client to return some layouts
> without specifying which ones.  CB_RECALL_ANY seemed custom-made for this
> purpose, so we used it.
> 
> I don't think it would be appropriate for the server to recall all layouts
> when it only needs some of them back.

As I said previously: the current client implementation deals with
CB_RECALL_ANY by calling initiate_file_draining(), which forgets _all_
layouts. If that is what we want to continue to use, then sending
layoutreturn with LAYOUTRETURN4_ALL is appropriate.
Otherwise, please fix the client to be more selective (in which case
LAYOUTRETURN4_FILE may be more appropriate) and please remember to
provide performance numbers to justify the need for optimisation.

NetApp

---

From: Jim Rees <[email-masked]>
Date: Mon, 31 Oct 2011 14:31:31 -0400

Trond Myklebust wrote:

---

From: "Welch, Brent" <[email-masked]>
Date: Mon, 31 Oct 2011 21:42:43 +0000

CB_RECALL_ANY was added based on Panasas experience with its own protocols.   When you have
100's (or 1000's) of clients and millions of files for which you maintain state, it is
very much worth optimizing how the server can reclaim this state.
	Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jim Rees
Sent: Monday, October 31, 2011 11:32 AM
To: Trond Myklebust
Cc: [email-masked]; nfsv4 list
Subject: Re: [nfsv4] [PATCH 2/2] nfs41: handle BLK_LAYOUT CB_RECALL_ANY

Trond Myklebust wrote:

I don't think it would be appropriate for the server to recall all layouts
when it only needs some of them back.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4