Subject: [nfsv4] (old) comments regarding draft-ietf-nfsv4-scsi-01

From: David Noveck <[email-masked]>
Date: Fri, 25 Sep 2015 15:30:00 -0400

I tried to review section 3 and was quite confused by it.  I realize that
it closely follows RFC5663 but that didn't help dispel my confusion.   I
think we have a problem in that those who have implemented pNFS block have
never clearly explained exactly how it is supposed to work.  The basic
problems that I saw in section 3 were:

   - A lack of clarity about whether security (on a per-principal basis) or
   locking (on a per-owner basis) was being enforced.
   - A lack of clarity about the respective roles of the MDS and client in
   making sure these constraints were enforced.
   - A lot of stuff that needed to be clearly explained before the Security
   Considerations section is left to be hurriedly explained there.

As an example of the kind of thing I'm talking about, consider the
following:

Similarly, SCSI storage devices are unable to validate NFS Access
Control Lists (ACLs) and file open modes, so the client must enforce
the policies before sending a READ or WRITE request to the storage
device.

Which has the following issues:

   - It's true that storage devices can't deal with ACL's but NFSv4 clients
   aren't supposed to. They are supposed to depend on the server to do this
   when using ACCESS and OPEN.
   - It isn't very clear what specific "policies" it is referring to.
   - If you are supposed to check this stuff before issuing the IO, what
   about the window between the check and the IO?

So here's a possible replacement to stimulate discussion.  It consists of a
new section (with some subsections) and  a new proposed
(very-much-shortened) Security Considerations section.  I'm not sure this
accurately describes current implementations but I think it is clear and
would work if implemented.  Please let me know if you disagree or have any
other comments.

I've built this possible replacement based on Tom's intuition, as stated in
draft-ietf-pnfs-types that it is the job of the client and data storage
device to enforce NFSv4 semantics.  Although Tom focuses on the
authorization issues only, we need to clearly extend that to deal with
locking restrictions as well.  The new section is built around that idea
and structures the responsibilities as follows:

   - the responsibility for making decisions as to what access is
   semantically valid is the server's.
   - the basic responsibility for enforcing those decisions is the pNFS
   clients'.
   - When situations make it impossible for the client to properly enforce
   those decisions, the servers may fence a client as a last resort in order
   to make sure that NFSv4 semantics is respected.

Note that in this version I've eliminated the material about virtualized
block and what it might be able to do.  This is not because I think it
should not be discussed.  It is just that I think we need to focus on
clearly describing the simple case first.  Later I'll have further comments
about virtualized block.

*2. Enforcing NFSv4 Semantics*

The functionality provided by SCSI Persistent Reservations makes it
possible for the MDS to control access by individual client machines
to specific LUs.  Individual client machines may be allowed to or
prevented from reading or writing to certain block devices.
Finer-grained access control methods are not generally available.

For this reason, certain responsibilities for enforcing NFSv4
semantics, including security and locking, are delegated to pNFS
clients when SCSI layouts are being used.  The metadata server's role
is to only grant layouts appropriately and the pNFS clients have to be
trusted to only perform accesses allowed by the layout extents they
currently hold (e.g., and not access storage for files on which a
layout extent is not held).  In general, the server will not be able
to prevent a client that holds a layout for a file from accessing
parts of the physical disk not covered by the layout.  Similarly, the
server will not be able to prevent a client from accessing blocks
covered by a layout that it has already returned.  The pNFS client
must respect the layout model for this mapping type to appropriately
respect NFSv4 semantics.

Furthermore, there is no way for the storage to determine the specific
NFSv4 entity (principal, openowner, lockowner) on whose behalf the IO
operation is being done.  This fact may limit the functionality to be
supported and require the pNFS client to implement server policies
other than those describable by layouts.

In cases in which layouts previously granted become invalid, the
server has the option of recalling them.  In situations in which
communication difficulties prevent this from happening, layouts may be
revoked by the server.  This revocation is accompanied by changes in
persistent reservation which have the effect of preventing SCSI access
to the LUs in question by the client.

*2.1 Use of Open Stateids*

The effective implementation of these NFSv4 semantic constraints is
complicated by the different granularities of the actors for the
different types of the functionality to be enforced:

   - To enforce security constraints for particular principals.
   - To enforce locking constraints for particular owners (openowners
and lockowners)

Fundamental to enforcing both of these sorts of constraints is the
principle that a pNFS client must not issue a SCSI IO operation unless
it possesses both:

   - A valid open stateid for the file in question, performing the IO
that allows IO of the type in question, which is associated with the
openowner and principal on whose behalf the IO is to be done.
   - A valid layout stateid for the file in question that covers the
byte range on which the IO is to be done and that allows IO of that
type to be done.

As a result, if the equivalent of IO with an anonymous or write-bypass
stateid is to be done, it MUST NOT by done using the pNFS SCSI layout
type.  The client MAY attempt such IO using READs and WRITEs that do
not use pNFS and are directed to the MDS.

When open stateids are revoked, due to lease expiration or any form of
administrative revocation, the server MUST recall all layouts that
allow IO to be done on any of the files for which open revocation
happens.  When there is a failure to successfully return those
layouts, the client in question MUST be prevented from any SCSI device
covered by these layouts until ...

*2.2 Enforcing Security Restrictions*

The restriction noted above provides adequate enforcement of
appropriate security restriction when the principal issuing the IO is
the same as that opening the file.  The server is responsible for
checking that the IO mode requested by the open is allowed for the
principal doing the OPEN.  If the correct sort of IO is done on behalf
of the same principal, then the security restriction is thereby
enforced.

If IO is done by a principal different from the one that opened the
file, the client SHOULD send the IO to be performed by the metadata
server rather than doing it directly to the storage device.

*2.3 Enforcing Locking Restrictions*

Mandatory enforcement of whole-file locking by means of share
reservations is provided when the pNFS client obeys the requirement
set forth in Section 2.1 above.  Since performing IO requires a valid
open stateid an IO that violates an existing share reservation would
only be possible when the server allows conflicting open stateids to
exist.

The nature of the SCSI layout type is such implementation/enforcement of
mandatory byte-range locks is very difficult. Given that layouts are
granted to clients rather than owners, the pNFS client is in no position to
successfully arbitrate among multiple lockowners on the same client. Suppose
lockowner A is doing a write and, while the IO is pending, lockowner B
requests a mandatory byte-range for a byte range potentially overlapping
the pending IO. In such a situation, the lock request cannot be granted
while the IO is pending. In a non-pNFS environment, the server would have
to wait for pending IO before granting the mandatory byte-range lock. In
the pNFS environment the server does not issue the IO and is thus in no
position to wait for its completion. The server may recall such layouts but
in doing so, it has no way of distinguishing those being used by lockowners
A and B, making it difficult to allow B to perform IO while forbidding A
from doing so. Given this fact, the MDS need to successfully recall all
layouts that overlap the range being locked before returning a successful
response to the LOCK request. While the ;lock is in effect, the server
SHOULD respond to requests for layouts which overlap a currently locked
area with NFS4ERR_LAYOUTUNAVAILABLE. To simplify the required logic a
server MAY do this for all layout requests on the file in question as long
as there are any byte-range locks in effect.

Given these difficulties it may be difficult for servers supporting
mandatory byte-range locks to also support SCSI layouts. Servers can
support advisory byte-range locks instead. The NFSv4 protocol currently has
no way of determining whether byte-range lock support on a particular file
system will be mandatory or advisory, except by trying operation which
would conflict if mandatory locking is in effect. Therefore, to avoid
confusion, servers SHOULD NOT switch between mandatory and advisory
byte-range locking based on whether any SCSI layouts have been obtained or
whether a client that has obtained a SCSI layout has requested a byte-range
lock.

*3. Security Considerations*

Access to SCSI storage devices is logically at a lower layer of the
I/O stack than NFSv4, and hence NFSv4 security is not directly
applicable to protocols that access such storage directly.  Depending
on the protocol, some of the security mechanisms provided by NFSv4
(e.g., encryption, cryptographic integrity) may not be available or
may be provided via different means.  At one extreme, pNFS with SCSI
layouts can be used with storage access protocols (e.g., parallel
SCSI) that provide essentially no security functionality.  At another
extreme, pNFS may be used with storage protocols such as iSCSI that
can provide significant security functionality.  It is the
responsibility of those administering and deploying pNFS with a SCSI
storage access protocol to ensure that appropriate protection is
provided to that protocol (physical security is a common means for
protocols not based on IP).  In environments where the security
requirements for the storage protocol cannot be met, pNFS SCSI layouts
SHOULD NOT be used.

When security is available for a storage protocol, it is generally at
a different granularity and with a different notion of identity than
NFSv4 (e.g., NFSv4 controls access by principals to  files, iSCSI
controls initiator access to volumes).  The responsibility for
enforcing appropriate correspondences between these layers is shared
by the metadata server and the pNFS client as described in Section
2.1. As with the issues in the first paragraph of this section, in
environments where the security requirements are such that
considerable client-side participation in enforcing security
restrictions is not acceptable, pNFS SCSI layouts SHOULD NOT be used.

To return to the subject of virtualized block for a bit, let me first
state my conclusions.

   - Virtualized block is a good thing to implement.  This is
particularly so now given the difficulties with RPC-RDMA (spec and
implementation).  It would be great to leverage SCSI's ability to do
DMA and use it in a file context.
   - The primary advantage of virtualized block is that it enables
much simpler layouts for the client to deal with;  a client can
receive a simple layout for the entire file simplifying his layout
management task.  In addition he should not have to deal with separate
read and write layouts for supporting snapshots/clones.
   - It would be helpful if the client knew he had a simpler layout
management task.  This would require either new information in the
device descriptions or a separate layout  type for virtualized block
   - An important advantage of virtualized  block is in the area of
safety (as opposed to security).  With virtualized block, a
client-side arithmetic error cannot wind up destroying a critical
piece of fs metadata.   Any arithmetic error is most likely to affect
the file in question, making the issues pretty much like those in
doing pNFS file.
   - Given that scsi knows only about clients (and not principals) I
don't see any way that we can totally avoid trusting the clients to
some degree.
   - It might be possible to tell clients that layouts to particular
devices only be used by particular principals (or principals in a
particular group).  That would mean that enforcement while still part
of the client could be done in smaller set of places.  This would also
require a new layout type.

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 29 Sep 2015 09:43:30 +0200

Hi David,

thanks for the feedback, and I plan to incorporate most of your
rewritten text into the document.

> not use pNFS and are directed to the MDS.

Two comments on this one:

First, why just open stateid?  To be coherent with other NFSv4 documents
this probably should read open, lock or delegation stateid.

Second this doesn't match existing implementations which are heavy on
use of special stateids.  I actually agree with you that pNFS client
implementations SHOULD NOT use special stateids for accessing files
stored on storage devices.

However this is an issue with all loosely coupled pNFS implementations
and thus affects the block layout, object layout, flex files layout, pNFS
layout considerations RFCs.  Looking back it probably should have been
in RFC5661.

For the SCSI layout draft I'd be happy add the above language in as a
SHOULD, but not a MUST based on this consideration, and really like to
see this consideration explained in more detail in a layout-agnostic
document.

> covered by these layouts until ...

Again this is a generic issue for any loosely coupled layout.  I also
actively disagree with this condition: given that we need an
open/lock/delegation stateid AND a layout stateid there is no need to
revoke the layout stateid when revoking the o/l/d stateid.

Moreover layout recalls are not based on recalling the particular stateid,
they recall all access for the given range or fsid, which will affect
other open stateids as well.  Based on that I don't think adding this
clause is a useful idea.

> server rather than doing it directly to the storage device.

As long as this remains a SHOULD I'm fine with this clause.  Common
client implementation do not always enforce this due to operating system
limits in the buffered writeback path, but they at least try hard.

Again this isn't an issue  specific to this layout type, but I'd be happy to
add it as it's simple and seems useful.

> *2.3 Enforcing Locking Restrictions*

<snip>

This is similar but different from the text we're proposing for flexfiles.
Can we agree on a common wording for both?

---

From: David Noveck <[email-masked]>
Date: Tue, 29 Sep 2015 10:29:27 -0400

> First, why just open stateid?

I'll explain in a bit.

> To be coherent with other NFSv4 documents

I believe that it is coherent with other documents.  It is just that it is
not the same as those other documents and I believe that is for good
reasons.

I believe those other documents deal with stateids that one presents in an
NFSv4
operation. On the other hand, this text deals with stateids that one must
posess
to assure that you have an open file that the MDS knows about.  Since, you
are
using an IO protocol that has no concept of stateids, the situation is
different
making the text seem discordant even though it isn't.

As far as lock stateids, they aren't mentioned because you can't have a
valid lock stateid
without a valid open stateid.  As far as delegation stateids, some
correction is
needed although it isn't simply a matter of allowing delegation stateids in
addition to
open stateids.  If you possess a write delegation, you can open a file
without contacting
the MDS and getting an open stateid, so the text needs to be updated to
reflect that
fact.

> use of special stateids.

I'm not sure exactly what you mean by "heavy on the use of special
stateids"
given that the actual requests sent do not contain stateids.  The text in
question refers, not to the stateids actually sent (since there aren't
any), but
to the stateids that the client actually possesses.  In other words, the
issue is
whether these clients are (knowingly or not) doing IO using pNFS on files
that
have not been opened on the MDS.  Let's put aside RFC2219 terms for a bit.
Can we agree that such clients are BROKEN?

> I actually agree with you that pNFS client

I didn't say that.

> layout considerations RFCs.  Looking back it probably should have been

Yes but, all of a sudden I don't feel so good.  I think this a case of
*RFC5661bis *
*Dysphoria* (not in the DSM).  Perhaps you should have put in a "trigger
warning" :-)

> Again this is a generic issue for any loosely coupled layout.

This applies to some layouts but not all.  Not sure how to deal with issues
that deal
with loosely  coupled layouts as a class.  Maybe this should be dealt with
in
draft-ietf-pnfs-layouts.

> I also

Actually there is a need.  The reason is that when you revoke an open, the
situation
is different from recalling a layout/delegation.  When you have a recall,
the client must
acknowledge the action, by returning the object in question.  In the case
of layouts, the
MDS has fencing available to make sure that the revocation is adhered to,
any communications
problems notwithstanding.   One option, which I didn't pursue because it
seemed too
complicated, would be to allow the MDS not to recall the layouts if the
client
acknowledged the revocation by using FREE_STATEID.  The MDS would have to
delay IO's
done by the client in question, from the period at which the revocation
happened
until it was acknowledged.

> other open stateids as well.

That's true.  However, those whose stateids are still OK, can reget the
layouts,
as long as fencing has not occurred.

> Based on that I don't think adding this
> clause is a useful idea.

I disagree.

> Can we agree on a common wording for both?

Probably.  I'm not sure where the flexfiles document is right now and what
the current relevant text is.
Maybe we should revisit this after flexfiles-07 is out.