Subject: [nfsv4] review of draft-ietf-nfsv4-flex-files-06 (part one of two)

From: David Noveck <[email-masked]>
Date: Thu, 3 Sep 2015 13:21:12 -0400

*General comments*

I agree with most of Benny's comments.  I'll try to avoid duplication and
only mention them when there is something about them that I disagree with.

*Overall Evaluation*

Overall, this spec, although it has made considerable progress, has a
significant set of issues that remain to be addressed.  It may take a while
to resolve these and get this document ready for presentation to the IESG.

*Summary of issues*

The only major technical issues that I've found are discussed in the part
of the review primarily devoted to reviewing the section *State and Locking
Models*. Because this section interacts with the section *ff_layout4*
(addressed in the second review part), I've decided to discuss the
potential interaction between these sections in separate sections (both in
this part of the revirew). These are in the sections *Stateid Choices*
and *Anonymous
IO Issuers.*

There's a significant administrative issue to be addressed, as described in
the section *CB_RECALL* (in the second part of the review).

Among the editorial issues, I'd mention the following as particularly
important:

   - Getting rid of "Data server".
   - Dealing with the last remnants of the no-control-protocol shtick.
   - Dealing with the issues in *S**t**ate and Locking Models* (may be
   somewhat less extensive if a decision is made to follow the basic approach
   laid out in *ff_layout4*).

*Comments by Section (Through Section 2.3):*

*Abstract:*

Suggest replacing "to allow" by "which allows"

*1. Introduction:*

Suggest replacing "the wire protocol of such a protocol" by "such a
protocol".

*1.1 Definitions:*

   - control protocol

This is not "a set of requirements".  Rather it is something that supports
a set of requirements.  Unstated requirements with nothing to provide
support them are not very helpful.

As an example of the difficulties that this gives rise to, consider the
phrase "requirements for the protocol" in the introduction.  If the
control protocol is a "set of requirements" then the requirements for the
protocol is "requirements for a set of requirements" :-(  This is making my
head hurt.

It's time to drive a stake through the heart of this.

   - data file

At the very least, should replace "E.g." by "I.e.".  my preference would be
to simply say it is the file contents without using any
Latin abbreviations.  So how about:

That part of a file system object that consists of the file contents.

   - data server (DS)

I think this definition should be deleted together with *section 1.2*.  See
my comments below for more details/ranting.

   - File layout type

I think you mean here the layout type defined in chapter 13 of RFC5661.  As
it is, with the current reference to 'the NFS protocol" this seems
to include that and the flexible file layout.   If you need a term for
that latter, suggest "NFS-based layout type".

   - layout segment

The reference there is to *chapter 13* of RFC5661 which deals with the
(old-fashioned,  tightly-coupled) file layout type.  It makes more sense,
in this document, to reference something specific to  the flex-files layout
type.

   - layout stateid

In the last sentence suggest adding "of that document" after "Section
12.5.3".

   - loose coupling

You seem to have gotten rid of the "no control protocol" stuff elsewhere.
 it's time to do the same in the definitions section.

   - metadata file

This is now confusing because, although the metadata file does
not include the file contents, it does describe them.  So "describes the
object and not the payload" is wrong.

Suggest the following replacement:

That part of a file system object which gives attributes for and the
location of the file data rather than containing the file contents itself,
 It can include such items as the times of  last modification or access,
and security information such as an ACL.

   - mirror

There are a couple of issues I have with the second sentence or the first
sentence following the initial sentence fragment):

   - It isn't clear to me how the two clauses relate.
      - there needs to be more information about how the situation referred
      to in the first clause is arrived.
      - None of this is really part of the *definition *of mirror.

Some of this material needs to go into the *Mirroring* section.

   - tight coupling

Same issue as for "loose coupling"  You need
to characterize the different types/strengths of potential control
protocols, rather than pretending that a very thin one doesn't exist.  One
might say, with apologies to Kant, "non-existence is not a predicate" :-)

   - recalling a layout

In the last sentence, suggest "could be able" by "has the opportunity".

Also, I'm not sure what "etc" is referring to in the last sentence.  Should
it be removed?

   - revoking a layout

It seems to me that, for this layout type, revoking a layout and fencing
are essentially identical and that this fact should be noted.

1.2. Difference Between a Data Server and a Storage Device

The term data server appears eleven times in this document.  Nine of
them are either in the definitions of "data server" and "data storage
device" or in this section.  That leaves two:  the title of section
2.2 and the corresponding TOC entry.

It's time to terminate this with extreme prejudice.

*2.1. LAYOUTCOMMIT:*

There are a couple of issues in the first sentence:

   - "tightly coupled *system" *is not as clear/general as it should be.
   - The reference to the semantics of the File Layout Type is
inappropriate given that the reference is to *Chapter 12 *of RFC5661.

Suggest the following replacement:

When tightly coupled storage devices are used, the metadata server has
the responsibility, upon receiving a LAYOUTCOMMIT (see Section 18.42
of [RFC5661] <https://tools.ietf.org/html/rfc5661#section-18.42>), of
ensuring that the semantics of pNFS are respected (see Section 12.5.4
of [RFC5661] <https://tools.ietf.org/html/rfc5661#section-12.5.4>).
These do not include a requirement that data written to data storage
device be stable upon completion of the LAYOUTCOMMIT.

It makes sense to start a new paragraph at this point.

With regard to the rest of material, I basically agree with Benny but
I think we have to arrange things to make clearer that we are talking
about the loosely coupled case here.  I think you need something like
the following:

In the case of loosely coupled storage devices, it is the
responsibility of the client to make sure the data file is stable
before the metadata server begins to query the storage devices about
the changes to the file. If any WRITE to a storage device did not
result with stable_how equal to FILE_SYNC, a LAYOUTCOMMIT to the
metadata server MUST be preceded by a COMMIT to the storage devices
written to.

The last sentence of the paragraph is not very convincing as to the reason
for this requirement.  It should explain why "the LAYOUTCOMMIT might
not be synchronized
to the last WRITE operation to the storage device".  I assume the worry is
about a data storage device reboot immediately after the LAYOUTCOMMIT.

2.2. Fencing Clients from the Data Server:

If one replaces "Data Server" by "Data Storage Device" in the title, one
gets rid of the last two uses of "Data server" in this document. Thus there
is no need to discuss the differences/nuances between the two.

There's a problem in the fourth paragraph. given that the first three
paragraphs deal with the loosely coupled case, when you get to the tightly
coupled case you need to say something about fencing clients from the data
server. Saying it's not your job is OK if you say whose job it is.

*2.3. State and Locking Models:*

This review section deals with issues internal to this section. Issues
regarding contradictions/interaction with *ff_layout4* are discussed in the
section *Stateid Choices* below.

This section needs a fair amount of work, Major problems are:

   - Although the introductory sentences mention OPEN, LOCK, DELEGATION,
   the rest of the discussion focuses on opens, leaving it very unclear how
   byte-range locks and delegations will/should/might be dealt with. I think
   this is primarily an editorial problem although there are potential
   interactions with choices regarding fundamental technical choices as far as
   NFSv4.x.
   - In the loose-coupling case, the three NFS protocols are treated as
   essentially the same, despite their very real differences. This is, in
   part, an editorial problem, but it appears to me that once the editorial
   problems are addressed, one faces significant technical issues, See below
   for details.

A relatively minor matter concerns the last paragraph. Since no actual
control protocol is "described" in RFC5661, I suggest changing "described
in" to "provided for in"

Although, I feel this section needs to be broken up into sub-sections, my
choice of subsections below is not intended as a guide to any eventual
redrafting.  It is merely a way of trying to organize my comments about the
issues that exist.

*Byte-Range Locks:*

The issues here are primarily editorial. I think what you have is on the
right track but you need to explicitly state some things: For v3, what to
state is clear, but for v4.x, what to state depends on how you address some
of the issues with v4.x clientids for locks in general.

   - That mandatory byte-range locks are not ("cannot be"?) supported in
   the loosely-coupled case
   - You have to deal with the issue of which client ids the resulting
   state ids are associated with. That will depend on the particular protocol
   you are dealing with and some choices that one might make regarding
   clientids for open as regards v4.x. For NFSv3 data storage devices, you can
   do the lock operations locally on the MDS, associate the stateids with the
   client's MDS clientid (there isnt any other one) and things will work.
   However, the spec needs to state this. The v4.x cases are more complicated
   :-(

*Delegations:*

Although the details might differ with the choices made, the spec would
have to make clear how delegation recall and return work.

Another issue that needs to be addressed is delegation scope. Although it
is desirable for the scope of delegation to be (real) clients, if the
client ids used on open are those of the MDS, then the the delegation will
be to the MDS. That would be inconsistent with the real client assuming he
had exclusive access to the file in question since other clients accessing
the same MDS might have access as well.

*NFSv3:*

Here the document needs to stop pretending that v3 is v4. It needs to stop
to stop talking about opens and stateids for a protocol that doesn't have
either. If using an anonymous stateid as ffds_stateid is intended, it needs
to deal with the issues that result. Note that the issues are the same ones
discussed regarding using an anonymous stateid for NFSv4 IO which is
discussed in *ff_layout4* (in part 2 of the review)

*NFSv4.x in General:*

There are some issues that need to be addressed given that you are dealing
with stateful protocols:

   - On what client's behalf (i.e with what clientid) are requests made.
   Some possibilities are:
      - the MDS might use the clientid which represents the client on the
      MDS,
      - it might use the clientid that represents the MDS (as a client
      doing opens) on the data storage device.

Note that both of these are likely to be different from the clientid that
represents the client on the data storage device.

Also note that openowners are implicitly qualified by the associated
clientid. If you have two openowners that happen to be associated with the
same string, then, if the MDS is using its own clientid to make open
requests, then two open from different clients might be merged into a
single stateid.

   - Issues that relate to loss of state when than the data storage device
   reboots.

*NFSv4.0:*

Although it needs to deal with the difficulties mentioned above, v4.0 has
an advantage over v4.1 in this regard. In v4.0 stateids are not tied to a
particular clientid. this makes it easier to have the MDS get a stateid on
behalf of a client and give it to the client for use.
* Loosely-coupled NFSV4.1:*

In v4.1 stateids are generally tied to clientids. So for a stateid to be
usable with clientid X it must have been gotten on a session with clientid
X and use on the same session or another session that is tied to the same
clientid. It is conceivably possible for this to happen but the protocol
has to have provision that allowed the clientid to be passed among
different clients (e.g. the real client, the MDS acting as a client on the
original client's behalf).

*Stateid Choices:*

There is a basic choice to be made between two approaches regarding how
stateid's are to be used for NFSv4 clients in the loosely-coupled case:

   - The approach currently used in *State and Locking Models* in which the
   stateids to be used by the client for IO are those obtained by the MDS
   (acting as an NFSv4.x client) doing an OPEN on the data storage device
   (acting as an NFSv4.x server).
   - The approach currently specified in *ff_layout4* in which the stateids
   to be used by the client for IO are always the anonymous stateid.

A few important things to take note of:

   - Although one might think that adopting the approach from
*ff_layout4* would
   avoid the technical issues in *State and Locking Models*, that would not
   be fully correct. The MDS would have to do opens and closes, and deal with
   locks and delegations. The only difference would be whether the pNFS client
   would use those stateids to do IO.

Given that the issue of stateid-clientid binding only applies to NFSv4.1,
it might be possible to adopt the *State and Locking Models*, for NFSv4.0
and the *ff_layout4* approach for NFSv4.1.

   - Many of issues discussed in the *Anonymous IO Issuers *section will
   apply to NFSv4.x data storage devices only if the *ff_layout4* approach
   is followed. However, they will all apply to NFSv3 data storage devices
   whichever of the two approaches is adopted.

*Anonymous IO Issuers:*

There are a number of situations in which pNFS flex-file clients will be
issuing IO with no indication which owner it is on behalf of:

   - Arises whenever NFSv3 is the protocol used to access the data storage
   device. Since NFSv3 is stateless, there are no openowners or lockowners,
   all IO is issued anonymously.
   - Arises in the case in which NFSv4.x is the protocol used to access the
   data storage device and the *ff_layout4 *approach is followed. In this
   case, pNFS IO to the data storage device would be issued using the
   anonymous stateid, so the data storage device has no knowledge of the
   associated openowner/lockowner.
   - Arises in the case in which NFSv4.x is the protocol used to access the
   data storage device, coupling is loose, and mandatory byte-range locking is
   in effect. This applies even in the case in which the *State and Locking
   Models* approach is used. in the loose coupling case, all pNFS flex-file
   IO's to the data storage device will be issued anonymously with regard to
   the identity of the associated lockowner

Issues will arise with regard to all forms of mandatory locking in NFSv4. e
will deal with each in turn.

With regard to mandatory byte-range locking, there are a number of choices:

   - Simply state that mandatory byte--range locking is not supported on
   server that support the pNFS flex-files mapping type.
   - Explain how it might be done, which involves forcing IO to go through
   the MDS whenever mandatory byte-range lock are in effect for a particular
   byte-range (or file).

With regard to mandatory locking due to share reservations one doesn't have
the option of simply not supporting the functionality. The spec will have
to clearly explain how it is to be done. Some likely elements:

   - In the case in which each of the clients with a particular file
   opened, has the same IO rights, the MDS has to ensure via layout recalls
   (and potential indicating layouts are unavailable) that no client which has
   no owner allowed to a particular form of IO has no layouts that allow that
   form of IO to be done. (it may already say that but it probably needs
   clarification).
   - In the case in which a particular client has multiple owners with
   different levels IO rights, the spec either has to ask the pNFS client to
   do the enforcement itself, or it has to provide that layouts are to be
   unavailable to this client and require the client to perform the IO via the
   MDS

---

From: Benny Halevy <[email-masked]>
Date: Sun, 6 Sep 2015 10:52:53 +0300

Thanks for the review Dave.
Some comments in line below.

On Thu, Sep 3, 2015 at 8:21 PM, David Noveck <[email-masked]> wrote:
> General comments
> Overall Evaluation
> Summary of issues
> The only major technical issues that I've found are discussed in the part of
> the review primarily devoted to reviewing the section State and Locking
> Models. Because this section interacts with the section ff_layout4
> (addressed in the second review part), I've decided to discuss the potential
> interaction between these sections in separate sections (both in this part
> of the revirew). These are in the sections Stateid Choices and Anonymous IO
> Issuers.
>
> the section CB_RECALL (in the second part of the review).
>
> Getting rid of "Data server".
> Dealing with the last remnants of the no-control-protocol shtick.
> Dealing with the issues in State and Locking Models (may be somewhat less
> extensive if a decision is made to follow the basic approach laid out in
> ff_layout4).
>
> Comments by Section (Through Section 2.3):
>
> Abstract:
>
> 1. Introduction:
> 1.1 Definitions:
> control protocol
> This is not "a set of requirements".  Rather it is something that supports a
> set of requirements.  Unstated requirements with nothing to provide support
> them are not very helpful.

Agreed.

> phrase "requirements for the protocol" in the introduction.  If the control
> protocol is a "set of requirements" then the requirements for the protocol
> is "requirements for a set of requirements" :-(  This is making my head
> hurt.
>
> data file
> to simply say it is the file contents without using any Latin abbreviations.
> So how about:
>
> data server (DS)
> I think this definition should be deleted together with section 1.2.  See my
> comments below for more details/ranting.
>
> File layout type
>
> it is, with the current reference to 'the NFS protocol" this seems to
> include that and the flexible file layout.   If you need a term for that
> latter, suggest "NFS-based layout type".
>
> layout segment
>
> The reference there is to chapter 13 of RFC5661 which deals with the
> (old-fashioned,  tightly-coupled) file layout type.  It makes more sense, in
> this document, to reference something specific to  the flex-files layout
> type.
>
> layout stateid
>
> loose coupling
> metadata file
> This is now confusing because, although the metadata file does not include
> the file contents, it does describe them.  So "describes the object and not
> the payload" is wrong.
>
> It can include such items as the times of  last modification or access, and
> security information such as an ACL.
>
>
> mirror
>
> It isn't clear to me how the two clauses relate.
> there needs to be more information about how the situation referred to in
> the first clause is arrived.
> None of this is really part of the definition of mirror.
>
> Some of this material needs to go into the Mirroring section.
>
> tight coupling
>
> Same issue as for "loose coupling"  You need to characterize the different
> types/strengths of potential control protocols, rather than pretending that
> a very thin one doesn't exist.  One might say, with apologies to Kant,
> "non-existence is not a predicate" :-)
>
> recalling a layout
>
> revoking a layout
> It seems to me that, for this layout type, revoking a layout and fencing are
> essentially identical and that this fact should be noted.
>
>
> The term data server appears eleven times in this document.  Nine of them
> are either in the definitions of "data server" and "data storage device" or
> in this section.  That leaves two:  the title of section 2.2 and the
> corresponding TOC entry.
>
>
> 2.1. LAYOUTCOMMIT:
> "tightly coupled system" is not as clear/general as it should be.
> The reference to the semantics of the File Layout Type is inappropriate
> given that the reference is to Chapter 12 of RFC5661.
>
> When tightly coupled storage devices are used, the metadata server has the
> responsibility, upon receiving a LAYOUTCOMMIT (see Section 18.42 of
> [RFC5661]), of ensuring that the semantics of pNFS are respected (see
> Section 12.5.4 of [RFC5661]).  These do not include a requirement that data
> written to data storage device be stable upon completion of the
> LAYOUTCOMMIT.
>
>
> With regard to the rest of material, I basically agree with Benny but I
> think we have to arrange things to make clearer that we are talking about
> the loosely coupled case here.  I think you need something like the
> following:
>
>
> In the case of loosely coupled storage devices, it is the responsibility of
> the client to make sure the data file is stable before the metadata server
> begins to query the storage devices about the changes to the file. If any
> WRITE to a storage device did not result with stable_how equal to FILE_SYNC,
> a LAYOUTCOMMIT to the metadata server MUST be preceded by a COMMIT to the
> storage devices written to.
>
>
> for this requirement.  It should explain why "the LAYOUTCOMMIT might not be
> synchronized to the last WRITE operation to the storage device".  I assume
> the worry is about a data storage device reboot immediately after the
> LAYOUTCOMMIT.
>
> 2.2. Fencing Clients from the Data
>
> Server:
>
>
> If o
>
> ne replaces "Data Server" by "Data Storage Device" in the title, one gets
> rid of the last two uses of "Data server" in this document. Thus there is no
> need to discuss the differences/nuances between the two.
>
> 2.3. State and Locking Models:
>
> regarding contradictions/interaction with ff_layout4 are discussed in the
> section Stateid Choices below.
>
> Although the introductory sentences mention OPEN, LOCK, DELEGATION, the rest
> of the discussion focuses on opens, leaving it very unclear how byte-range
> locks and delegations will/should/might be dealt with. I think this is
> primarily an editorial problem although there are potential interactions
> with choices regarding fundamental technical choices as far as NFSv4.x.
> In the loose-coupling case, the three NFS protocols are treated as
> essentially the same, despite their very real differences. This is, in part,
> an editorial problem, but it appears to me that once the editorial problems
> are addressed, one faces significant technical issues, See below for
> details.
>
> Byte-Range Locks:
> That mandatory byte-range locks are not ("cannot be"?) supported in the
> loosely-coupled case
> You have to deal with the issue of which client ids the resulting state ids
> are associated with. That will depend on the particular protocol you are
> dealing with and some choices that one might make regarding clientids for
> open as regards v4.x. For NFSv3 data storage devices, you can do the lock
> operations locally on the MDS, associate the stateids with the client's MDS
> clientid (there isnt any other one) and things will work. However, the spec
> needs to state this. The v4.x cases are more complicated :-(
>
> Delegations:
>
> Although the details might differ with the choices made, the spec would have
> to make clear how delegation recall and return work.
>
> Another issue that needs to be addressed is delegation scope. Although it is
> desirable for the scope of delegation to be (real) clients, if the client
> ids used on open are those of the MDS, then the the delegation will be to
> the MDS. That would be inconsistent with the real client assuming he had
> exclusive access to the file in question since other clients accessing the
> same MDS might have access as well.
>
> NFSv3:
>
> discussed in ff_layout4 (in part 2 of the review)
>
> NFSv4.x in General:
>
> On what client's behalf (i.e with what clientid) are requests made. Some
> possibilities are:
>
> the MDS might use the clientid which represents the client on the MDS,
> it might use the clientid that represents the MDS (as a client doing opens)
> on the data storage device.
>

If the client is going to use different clientid's for accessing the MDS and
the DS, it will break any attempt by the MDS to manipulate the client
state on the DS
via NFSv4+ or a backend protocol as the resulting stateid that is handed out
to the client with the layout, to be used for I/O requests against the
DS is qualified
by a clientid state "namespace" (see section 8.2 in RFC5661).

> requests, then two open from different clients might be merged into a single
> stateid.
>

ditto

> Issues that relate to loss of state when than the data storage device
> reboots.
>
>
> NFSv4.0:
>
> Although it needs to deal with the difficulties mentioned above, v4.0 has an
> advantage over v4.1 in this regard. In v4.0 stateids are not tied to a
> Loosely-coupled NFSV4.1:

Do we really need to support NFSv4.0 as an I/O protocol for tightly
coupled systems?
Since there is a requirement for the MDS and DSs to share a control protocol
that is a superset of what is specified in RFC5661 it is fair to assume that the
DS supports NFSv4.1 and possibly newer version.  It will make everybody's lives
easier if we just drop this from the version compatibility matrix.

> usable with clientid X it must have been gotten on a session with clientid X
> and use on the same session or another session that is tied to the same
> clientid. It is conceivably possible for this to happen but the protocol has
> to have provision that allowed the clientid to be passed among different
> clients (e.g. the real client, the MDS acting as a client on the original
> client's behalf).

agreed.

> Stateid Choices:
> The approach currently used in State and Locking Models in which the
> The approach currently specified in ff_layout4 in which the stateids to be
> used by the client for IO are always the anonymous stateid.

That is in the loosely coupled model.  Quoting:

   For tight coupling, ffds_stateid provides the stateid to be used by
   the client to access the file.  For loose coupling and a NFSv4
   storage device, the client may use an anonymous stateid to perform I/
   O on the storage device as there is no use for the metadata server
   stateid (no control protocol).  In such a scenario, the server MUST
   set the ffds_stateid to be zero.

> Although one might think that adopting the approach from ff_layout4 would
> avoid the technical issues in State and Locking Models, that would not be
> fully correct. The MDS would have to do opens and closes, and deal with
> Given that the issue of stateid-clientid binding only applies to NFSv4.1, it
> might be possible to adopt the State and Locking Models, for NFSv4.0 and the
> ff_layout4 approach for NFSv4.1.
>
> Many of issues discussed in the Anonymous IO Issuers section will apply to
> NFSv4.x data storage devices only if the ff_layout4 approach is followed.
> However, they will all apply to NFSv3 data storage devices whichever of the
> two approaches is adopted.
>
> Anonymous IO Issuers:
>
> Arises whenever NFSv3 is the protocol used to access the data storage
> device. Since NFSv3 is stateless, there are no openowners or lockowners, all
> IO is issued anonymously.
> Arises in the case in which NFSv4.x is the protocol used to access the data
> storage device and the ff_layout4 approach is followed. In this case, pNFS
> IO to the data storage device would be issued using the anonymous stateid,
> so the data storage device has no knowledge of the associated
> openowner/lockowner.
> Arises in the case in which NFSv4.x is the protocol used to access the data
> storage device, coupling is loose, and mandatory byte-range locking is in
> effect. This applies even in the case in which the State and Locking Models
> approach is used. in the loose coupling case, all pNFS flex-file IO's to the
> data storage device will be issued anonymously with regard to the identity
> of the associated lockowner

As mentioned earlier in your review, mandatory byte range locking cannot be
supported with the loose coupling model.

> Simply state that mandatory byte--range locking is not supported on server
> that support the pNFS flex-files mapping type.
> Explain how it might be done, which involves forcing IO to go through the
> MDS whenever mandatory byte-range lock are in effect for a particular
> byte-range (or file).

The assumption is that mandatory byte range locking can be done over the
flex-files layout with the tight coupling model using the backend protocol
to enforce such locks that are acquired via the MDS.  The means to do that
are out of scope for the layout type, though guidelines and examples can be
provided.  Though, as I said before, it would be even more beneficial
for potential
implementers to describe a compliant backend control protocol in a form
of an informational RFC or even a protocol standard, if of course
there is enough
interest to implement such a protocol so it progresses through the IETF's
standardization process.

Benny

> the option of simply not supporting the functionality. The spec will have to
> clearly explain how it is to be done. Some likely elements:
>
> In the case in which each of the clients with a particular file opened, has
> the same IO rights, the MDS has to ensure via layout recalls (and potential
> indicating layouts are unavailable) that no client which has no owner
> allowed to a particular form of IO has no layouts that allow that form of IO
> to be done. (it may already say that but it probably needs clarification).
> In the case in which a particular client has multiple owners with different
> levels IO rights, the spec either has to ask the pNFS client to do the
> enforcement itself, or it has to provide that layouts are to be unavailable
> to this client and require the client to perform the IO via the MDS
>
>