Subject: [nfsv4] pNFS commits with cluster file systems

From: Garth Goodson <[email-masked]>
Date: Wed, 22 Mar 2006 16:18:12 -0800

A discussion arose at the IETF meeting today regarding commits and 
cluster file systems.  I'll try to summarize:

NOTE: this is only relavent to the file-layout type.

The Problem
-----------
Some background first... In a cluster file system metadata must be 
committed through a single 'metanode'.  Thus, each NFSv4 COMMIT 
operation to a data server (DS) triggers a commit message to this 
metanode to commit the data and the portion of the metadata that pertain 
to that DS.  The current pNFS draft requires that COMMITs go to each 
data server that holds dirty data and that they maintain NFSv4 COMMIT 
semantics (i.e., they commit both data and metadata).

Marc Eshel has pointed out that this has a dire performance impact for 
cluster file systems: the DS COMMITs trigger commits to the metanode and 
may result in 2 I/Os being performed (one to write out dirty data, one 
for the metanode to write out the metadata describing that data).  The 
multiple I/Os for the metadata (one from each DS) can be avoided through 
a single commit to the metanode once all the data has been synced. 
Ultimately, these systems would like a single commit to the metanode to 
commit all dirty data.  (The current model is slightly backwards in that 
the DSs tell the metanode when to commit rather than the other way around).

Possible Solutions
------------------

1)
Marc initially proposed a 'data only' commit be sent to the DSs and 
another commit be sent to the metadata server (MDS), to sync the 
metadata, once the DS commits have completed.  This would batch the, 
possibly many, I/Os that would otherwise be necessary to commit the 
metadata into a single I/O.  He proposed using LAYOUTCOMMIT as the 
'metadata commit'.  This alters pNFS's current behavior, since the 
client would have to keep the data around until the MDS commit (i.e., 
LAYOUTCOMMIT) completed.  Marc proposed that these special commits only 
be done when the client knows that it is going to be following data 
commits with a LAYOUTCOMMIT (e.g., at a fsync call), thus not 
introducing more LAYOUTCOMMIT operations.  However, this introduces more 
complexity into the client.  As well, the separate metadata and data 
commits complicate recover.

2)
Marc, Manoj, Andy, and I talked today and came up with a solution that I 
prefer.  First, we must acknowledge that two types of file layouts 
exist: 1) those that require/prefer commits to each DS; 2) those that 
require/prefer a single commit to the MDS.  The cluster file system 
described above fits into (2).  This information must be made available 
to the client, probably through a flag in the file-layout structure.

Upon receiving a layout with this flag set (COMMIT_THROUGH_MDS), the 
client should not issue COMMITs to the DSs.  Instead it must issue a 
special commit (we can talk about what exactly to use, but lets think of 
it as a new op for now), to the MDS when it is ready to commit the dirty 
data.  Once the data is committed (i.e., the new MDS commit completes) 
the client can discard the data.

If this new MDS commit fails, the client must rewrite the dirty data it 
holds.  Most likely the layout pertaining to the file (region) will be 
recalled.  The client can either write the data through the MDS or it 
can obtain a new layout and use that to commit the data.

This solution avoids some of the recovery issues that occur with 
solution 1, it avoids the client sending different types of commits to 
DSs, and it feels more like the current v4 commit model.

-Garth

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Wed, 22 Mar 2006 19:38:03 -0800

One addendum to my last post on this subject.

In order to properly recover from server failures, nfsv4 relies on 
COMMITs validating a verifier passed into COMMIT which was generated by 
the WRITE.

The solution used by a cluster file system, must fail a MDS commit if it 
detects that a DS holding an unstable write has failed (and rebooted). 
To do this, the implementation is most likely going to require some 
state associated with the write (for NFS this is the write verifier) to 
be passed into the commit for verification.  A proposal for a solution 
to the cluster FS commit must address the problem solved by verifiers.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Sat, 25 Mar 2006 17:05:17 -0800

This note is to summarize the agreement we had at the IETF regarding an 
alternative way to commit data that should improve performance for 
cluster file systems. This applies to file layouts only.

A new flag in the file layout (COMMIT_THROUGH_MDS) will allow the MDS to 
tell the client of a preferred way of doing COMMIT. If this flag is set, 
the client SHOULD send COMMIT to the MDS instead of sending it to the 
same DS to which the associated WRITEs were sent. In order to maintain 
the current NFSv4 commit and recovery model, all the DS'es MUST return 
the same verifier for all WRITEs in a given file layout. Whenever any DS 
that is referenced in the layout reboots, the verifier MUST be changed. 
The single COMMIT to the MDS will return a verifier and the client must 
compare it to all the verifiers from the WRITEs and fail the COMMIT if
there is any mismatched verifiers. If COMMIT to the MDS fails, the client 
MUST reissue WRITEs for all the dirty data in the file as follows. If the 
layout has not been recalled, all WRITEs that have a different verifier 
than the one returned by COMMIT MUST be rewritten through the DS. If the 
layout has been recalled, the client MUST write the data through the MDS.

If there are no new objections than I will put in a format more 
appropriate for the draft.

Marc.

---

From: Benny Halevy <[email-masked]>
Date: Sun, 26 Mar 2006 16:43:18 +0200

> appropriate for the draft.

This looks good to me.

Benny

Marc Eshel wrote:
>>>metanode to commit the data and the portion of the metadata that 
> 
> pertain 
> 
>>>cluster file systems: the DS COMMITs trigger commits to the metanode 
> 
> and 
> 
>>>multiple I/Os for the metadata (one from each DS) can be avoided 
> 
> through 
> 
>>>Ultimately, these systems would like a single commit to the metanode 
> 
> to 
> 
>>>commit all dirty data.  (The current model is slightly backwards in 
> 
> that 
> 
>>>the DSs tell the metanode when to commit rather than the other way 
> 
> around).
> 
>>>Possible Solutions
>>>------------------
>>>
>>>1)
>>>LAYOUTCOMMIT) completed.  Marc proposed that these special commits 
> 
> only 
> 
>>>introducing more LAYOUTCOMMIT operations.  However, this introduces 
> 
> more 
> 
>>>Marc, Manoj, Andy, and I talked today and came up with a solution that 
> 
> I 
> 
>>>described above fits into (2).  This information must be made 
> 
> available 
> 
>>>special commit (we can talk about what exactly to use, but lets think 
> 
> of 
> 
>>>it as a new op for now), to the MDS when it is ready to commit the 
> 
> dirty 
> 
>>>If this new MDS commit fails, the client must rewrite the dirty data 
> 
> it 
>