Subject: [nfsv4] Reclaim bit for pNFS layout operations

From: [email-masked]
Date: Fri, 6 Jan 2006 16:10:07 -0500

Hi all---
This email is to summarize the discussion about the reclaim bit
for pNFS layout operations that occurred on the NFSv4.1 con call
of Jan 5, and to kick-start some in-depth technical discussion on
the subject.  Since there were a lot of people on the call, and
I don't know people's voices yet, I don't have proper attribution
for all of the stuff below.  So if I end up mis-representing
something you said, please jump in and clear the record...

Background:
pNFS GETLAYOUT requests do not currently have a
reclaim bit.  This means that when the server crashes, a client
cannot use the recovery grace period to reclaim a writable layout
that it previously held.  This may create a problem for a client
that has performed writes to a provisionally allocated range in a
file but has not yet updated the meta-data server with a LAYOUTCOMMIT
operation at the time that the meta-data server crashes.  The result
could be that the data written by the client is lost, and the
application experiences data corruption.

In fact, the problem is more severe in the case of the block/volume
layout than for file or object layouts.  In the file or object case,
it should be possible for the client to recover, in the worst case,
by re-reading the data from storage and writing it again after the
meta-data server recovers.  But for a block/volume client, if the
data has been written to a physical SAN location that was
provisionally allocated, once the meta-data server crashes and
recovers, there is no guarantee that the data at that location is
preserved.  It cannot be safely sourced by the client, and the only
solution available to the client is to hold each dirty data page
buffered in memory until a LAYOUTCOMMIT operation has been successfully
completed.

Requirement:
At a high level, it would be nice if the client had a way
of reliably reclaiming a portion of a previously-held writable layout
after the meta-data server crashes and recovers.  For the block/volume
layout, we have an additional requirement:  the physical block mapping
of the portion of the layout that is reclaimed should exactly match
the physical block mapping of the corresponding portion of the layout
that was held before the crash.

A few approaches were discussed, which I believe break down generally
into two options:

Option 1 (reclaim bit on LAYOUTGET):
It was suggested during discussion that adding a "reclaim bit" to
the pNFS LAYOUTGET operation provides a natural solution to the problem
above.  During the server recovery grace period, a client would be
allowed to send a LAYOUTGET request to the server for a range covering
the un-committed portion of the file, with the reclaim bit set.  Any
request during the recovery grace period that does not have the reclaim
bit would be rejected.  If the reclaim bit of a LAYOUTGET request is
set, the server would return the appropriate layout.  For the block/
volume case, the server would be required to ensure that the physical
block mapping for the provisional allocations in the layout is unchanged
from the layout held previously.  This requires the server to persistently
record the physical block mappings of provisional allocations.  For
block/volume layouts, if the server is incapable of ensuring this condition,
it must reject the LAYOUTGET operation when the reclaim bit is set.
The client MUST NOT set the reclaim bit on a LAYOUTGET request for a new
range.  That is, if the client sets the reclaim bit, the request MUST be
for a region of the file on which it held the same type of layout
immediately prior to the server crash.

Option 2 (separate LAYOUTRECLAIM operation):
Option 1 above requires the server to persistently record provisional
allocations for block/volume layouts.  Alternatively, during the recovery
grace period, the server could allow the client to supply the physical
mapping that it previously held when reclaiming the layout.  Since the
LAYOUTGET request args don't contain an appropriate data structure to
hold the physical mapping information, a new LAYOUTRECLAIM operation could
be added.  This operation would only be accepted by the server during the
recovery grace period.  It would function exactly like a LAYOUTGET operation
but would carry with it the physical mapping of the layout that the client
previously held.  This physical mapping would be used by the server for
any provisional allocations required for the layout.  The same client
restrictions as option 1 apply.  The server MUST NOT allow any block
allocation operations during the recovery grace period with the exception
of LAYOUTRECLAIM operations. 

It was noted in discussion that this approach would require a significant
level of trust to be invested in the client by the server.  It was further
noted that the server could apply some level of sanity checking on the
physical mapping provided by the client.  For instance, at a basic level
the server could verify that the blocks included in the client mapping
which correspond to provisional allocations are, in fact, free blocks in
the file system.  If the sanity checks fail, the server could reject the
LAYOUTRECLAIM operation.

Quick summary of the options:

Option 0 (status quo):
pros---Simplicity.  Minimal investment required.  ;)
cons---Potentially serious performance problems for block/volume layouts
(performance limited by amount of memory cache available to pNFS)

Option 1:
pros---Natural extension of v4 recovery approach.
cons---Requires persistent provisional allocations on server.  High bar
for server requirement.

Option 2:
pros---Relax requirements on server for recovery.
cons---New operation needed in protocol.  More trust invested in client.  

That's about it.
Corrections?  Comments?

-Steve

------------------------------------------------------------
Stephen Fridella                            ph: 508-305-8512
Principal Software Engineer                fax: 508-305-4803
EMC NAS Advanced Development         email: [email-masked]

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 6 Jan 2006 16:42:57 -0500

> the file system.  If the sanity checks fail, the server could reject the

So it doesn't look like option 2 *requires* the server to invest more
trust in the client--the server could still store the whole physical
mapping if it wanted to.  Option 2 just leaves the decision about how
much to trust the client up to the implementation.  Right?

--b.