Subject: RE: [nfsv4] Reclaim bit for pNFS layout operations

From: [email-masked]
Date: Fri, 6 Jan 2006 16:48:09 -0500

That's a fair statement.  Put another way, option 2 opens the
door for implementations that invest more trust in the client.
-Steve 

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Friday, January 06, 2006 4:43 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Reclaim bit for pNFS layout operations

On Fri, Jan 06, 2006 at 04:10:07PM -0500, [email-masked] wrote:
> It was noted in discussion that this approach would require a significant
> level of trust to be invested in the client by the server.  It was further
> noted that the server could apply some level of sanity checking on the
> physical mapping provided by the client.  For instance, at a basic level
> the server could verify that the blocks included in the client mapping
> which correspond to provisional allocations are, in fact, free blocks in
> the file system.  If the sanity checks fail, the server could reject the
> LAYOUTRECLAIM operation.

So it doesn't look like option 2 *requires* the server to invest more
trust in the client--the server could still store the whole physical
mapping if it wanted to.  Option 2 just leaves the decision about how
much to trust the client up to the implementation.  Right?

--b.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked]
Date: Tue, 10 Jan 2006 16:43:03 -0500

From: Garth Goodson [mailto:[email-masked]] 
> Halevy, Benny wrote:
> >>From the options you described I like option 2 the most as the server
can
> > always verify the layout reclaimed by the client against sanity rules or
> > even against a persistently stored copy of the layout (like the one
required
> > for option 1).  My question is whether this can be achieved without a
new
> > operation but rather with LAYOUTCOMMIT with a reclaim bit.
> > 
> > In the recovery grace period the server would allow only LAYOUTCOMMITs
> > with the reclaim bit set and will disallow any LAYOUTGETs.
> > 
> > Benny
> 
> I was thinking of this too.  At least in LAYOUTCOMMIT you have the 
> layoutupdate field which can be used by the client to specify what it 
> thinks the layout should be.  However, I think this might lead to a 
> slight overloading of semantics.  The client is committing a layout it 
> had, without getting a new layout.  My question is, what is the state of 
> the layout on the client after the LAYOUTCOMMIT succeeds?  Does it hold 
> the layout it just committed?
> 
> I guess the other alternative is a LAYOUTGET w/reclaim bit set, followed 
> (immediately) but a LAYOUTCOMMIT for that layout (with the client's 
> version thereof).
> 
> -Garth
> 

One approach is to identify the bare minimum semantics required
for recovery and provide only that.  For proper recovery, the
client just needs to commit the portions of the layout which cover
the uncommitted writes.  It does not *need* to reclaim a layout
for those blocks during the grace period---if it successfully commits
the blocks, it can simply wait for the grace period to expire and
request layouts for the file in the usual way.

Under this approach, the LAYOUTCOMMIT w/ reclaim bit set will
commit the LAYOUT if it succeeds, but it does not grant the layout
to the client.  Indeed, under this approach, no layouts are
granted to the client during the grace period.

I see a possible problem with the: LAYOUTGET w/ reclaim bit set
followed by a LAYOUTCOMMIT idea.  What if the LAYOUTGET succeeds
during the grace period, but the client delays the LAYOUTCOMMIT
for some reason.  How long should the server wait?  If the grace
period expires before the LAYOUTCOMMIT arrives, it may be impossible
for the server to honor the COMMIT, due to blocks in the layout
being re-allocated for other writes.  Then the client holds a
layout which it cannot commit---a very strange and unpleasant
state!

-Steve

---

From: [email-masked]
Date: Wed, 18 Jan 2006 15:07:45 -0500

Ok, its been a week, so let me try to summarize where
we are on this topic:

Originally two options were on the table...

Option 1---allow client to reclaim pNFS layouts during grace
period.  Server will persistently keep the pre-allocation
and guarantee that the same physical layout (for block/volume
case) will be returned to client upon reclaim.

Option 2---allow client to reclaim pNFS layouts during grace
period.  Also allow client to provide the physical layout
that it previously held.  Does not *require* the server to
keep pre-allocation persistent.  Server can sanity-check
the client provided layout and reject if it does not meet
safety conditions.  In the limit case server can make pre-
allocation persistent and reject the client's reclaim if
the client's mapping does not match the server's saved
mapping.

I did not see any enthusiasm for option 1 during the sub-
sequent list discussion.  The responses I saw all suggested
that option 2 was a better option, and focused on how to
make option 2 work.

Three approaches were suggested:

Approach 1:  A new LAYOUTRECLAIM operation which can only
succeed during the grace period.  The operation provides the
same parameters as LAYOUTGET, with the addition of a layout
update field which the client uses to provide what it
believes the mapping should be.

Approach 2:  Add a reclaim bit to the LAYOUTCOMMIT operation.
The reclaim bit can be used during the grace period only.
The client uses the layout update field to tell the server
its view of the pre-allocation mapping and to commit any
uncommitted blocks to the file.  This operation does not
obtain a layout for the client, it just allows it to commit
parts of the layout it previously held.

Approach 3:  Add a reclaim bit to the LAYOUGET operation,
and make the recovery process a two-operation sequence of
LAYOUTGET followed by LAYOUTCOMMIT where the LAYOUTCOMMIT
op contains the client's mapping.

Most of the responses seemed to favor approach 2.  Approach
1 got little support.  Approach 3 was mentioned in passing,
and I believe there are some tricky races associated with it.

I personally think that approach 2 is the simplest and most
workable.

Other views or ideas?

In the meantime, I'll work on some candidate language
describing approach 2 and post it for review...

-Steve

------------------------------------------------------------
Stephen Fridella                            ph: 508-305-8512
Principal Software Engineer                fax: 508-305-4803
EMC NAS Advanced Development         email: [email-masked]

-----Original Message-----
From: Garth Goodson [mailto:[email-masked]] 
Sent: Monday, January 09, 2006 1:48 PM
To: Halevy, Benny
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Reclaim bit for pNFS layout operations

Halevy, Benny wrote:
>>From the options you described I like option 2 the most as the server can
required
> for option 1).  My question is whether this can be achieved without a new

-Garth

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Friday, January 06, 2006 23:48
> To: [email-masked]; [email-masked]
> Cc: [email-masked]
> Subject: RE: [nfsv4] Reclaim bit for pNFS layout operations
>

---

From: "Halevy, Benny" <[email-masked]>
Date: Wed, 18 Jan 2006 15:27:32 -0500

> I personally think that approach 2 is the simplest and most

You got my vote on this of course :)

Benny

> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, January 18, 2006 22:08
> To: [email-masked]; [email-masked]