Subject: [nfsv4] Access permission issues in pNFS

From: sfaibish <[email-masked]>
Date: Thu, 18 Jun 2009 10:26:21 -0400

During the preparations for the Bakeathon and the interop
tests I realized that there is a possible gap in the pNFS
protocol regarding permissions of access to storage devices.
The question was what happens to a client that has no
permission to access a storage device. From the pNFS
protocol perspective there is no error mechanism to inform
that a client doesn't have the access permission to a
storage device at mount time. This is also the case with
the MDS that doesn't have access permission to some storage
devices.

This is true for file and object layout pNFS clients
regardless if the MDS has permission to access the storage
device. On the other hand looking at the block layout if the
MDS doesn't see the LUNs it cannot mount the pNFS file system
and of course it cannot allow to mount it and an error is
logged by the server. If the pNFS block server can access the
LUNs but the client doesn't have the access permission to
the LUN the client will send an error to the server and it
will mount as NFSv4.1 without pNFS support (fallback to NFS)
and the error is logged both at the client and the server.
If the client doesn't have access permission to all the
storage devices it will log an error at the client and at
the server. In this way an admin can take corrective
measures on either server or client or both.

Looking at the file and object layouts my understanding is
that there is no similar error mechanism to report the case
when the client or the server cannot access a storage device.
The only fallback that I see is a request for re-direct by
the MD server as storage device is unaccesible assuming that
the MD server has access to the storage device and it can
serve the I/O to the client still without logging an error
at least not at mount time. In the usecase when a new storage
device is added and all the clients (1000s of them) have no
access permission to the new storage device from this time
on all the IOs to the new storage device will be served by
the MD server creating a performance and scalability
bottleneck without possibility of detection.

After several discussions with Dave Black and Mike Eisler we
agreed that this is a possible gap in the protocol and the
server should have made sure the client had access to the
storage devices (files, luns, or objects). The sooner the
better, i.e. at mount time not at I/O time before the server
grants a layout to the client. We also agreed that there are
some difficulties to detect this situation at mount time in
cases of complex mountpoint structures and we can perhaps
only address the error for the root/top of the mount
structure assuming we are only refering to pNFS capable
clients. Although the change to the protocol is delicated
logging some kind of warning at the client might be
appropriate to be recommended as an implementation option
on the client.

My point being that we might need to enhance the error
mechanism at both server and client to address the above
issues. One of the problems may be the fact that there is no
mention in the spec to address the protocol between MDS and
storage device, except for the block layout driver. We are
also aware that an MDS that access the data stores is an
unhealthy pNFS server. Additional if we agree on an error
mechanism it might be better to be used at mount time than
at I/O time and perhaps to be logged by the server and the
client in either case.

So, to summarize (Dave wording) our preliminary discussions
we defined a scenario of interest or usecase and possible
client behavior to address this missing piece of protocol:

Scenario of interest: Client expects to be able to use pNFS
but one or more storage devices are inaccessible.  This
discussion does not apply to a client that doesn't care
(e.g., uses pNFS to optimize if available, but is ok if all
of its access is via the main NFS server).

Desired client behavior: Client gets entire device list for
mount point from server and checks it as part of the mount
operation (or at whatever point it first realizes that it
expects to use pNFS).

Missing piece of protocol: Client has no obvious way to report
an inaccessible device to the server.

The constraint for now: No functionality change to pNFS 4.1.

Implied Requirement: Metadata server SHOULD NOT use devices
in pNFS layouts that are not accessible to the MDS (or to
clients if the MDS has any means of determining this).

We are opening the discussion on possible solutions like
adding a new layout return type (I will leava this to Dave to
ellaborate on this). The intention should be not to change 4.1
(no doc respin) and to wait for 4.2 and add new method.

Your thoughts?

/Sorin

---

From: Jason Glasgow <[email-masked]>
Date: Thu, 18 Jun 2009 10:53:41 -0400

Yes, better reporting would be a good goal.  The object layout has defined a
object layout specific lrf_body to allow additional error information to be
communicated to the server.  The file and block layout could follow this
approach, or we could work towards something that applies to all layout
types.  My initial feeling is that this is best addressed in the layout
specific layer.

http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-pnfs-obj-12.txt

>  If the lora_layout_type layout type is LAYOUT4_OSD2_OBJECTS, then the
>    lrf_body opaque value is defined by the pnfs_osd_layoutreturn4 type.
>
>    The pnfs_osd_layoutreturn4 type allows the client to report I/O error
>    information back to the metadata server as defined below.
>
> 8.1.  pnfs_osd_errno4
>
>    /// enum pnfs_osd_errno4 {
>    ///     PNFS_OSD_ERR_EIO            = 1,
>    ///     PNFS_OSD_ERR_NOT_FOUND      = 2,
>    ///     PNFS_OSD_ERR_NO_SPACE       = 3,
>    ///     PNFS_OSD_ERR_BAD_CRED       = 4,
>    ///     PNFS_OSD_ERR_NO_ACCESS      = 5,
>    ///     PNFS_OSD_ERR_UNREACHABLE    = 6,
>    ///     PNFS_OSD_ERR_RESOURCE       = 7
>    /// };
>    ///
>
>    pnfs_osd_errno4 is used to represent error types when read/write
>    errors are reported to the metadata server.  The error codes serve as
>    hints to the metadata server that may help it in diagnosing the exact
>    reason for the error and in repairing it.
>
>
See additional comment below.

> the MDS that doesn't have access permission to some storage

I am a little confused about what you mean about an "MDS" that doesn't have
access to some storage devices.  The protocol between the MDS and the
storage devices has always been out of scope, and I believe it should stay
that way.

> bottleneck without possibility of detection.

The NFS protocol should not be involved with dicating behavior when the
server cannot access its storage devices.  This is not a protocol issue.

> appropriate to be recommended as an implementation option

My view here is somewhat block specific here, but the concept generalizes.
Accessiblity, but not readability or writability, should be ensured as early
as possible (at mount time is preferable), but because layouts are an
explicit grant of read or write access to ranges of the storage device, one
should not state that the entire device is anything more than
accessible/identifiable/addressable at mount time.  Ideally, access would
only be granted to regions of the storage device as layouts are given to
clients.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: sfaibish <[email-masked]>
Date: Thu, 18 Jun 2009 13:46:35 -0400

> types.  My initial feeling is that this is best addressed in the layout
Correct me if I am wrong but these errors are only refer to read/write
at I/O time.

>>    reason for the error and in repairing it.
I saw this and infact my initial observations were for file layout only
and I extend them to OSD. Still at mount time there is no error that can be
reported as these are I/O time errors and at mount time there is no access
check or I/O requests. And there is no specific error as permission access
is denied unless you use UNREACHABLE. I would ask Benny's opinion as well.

> storage devices has always been out of scope, and I believe it should  
I understand your confusion so let me try an example:
1. client get a layout from the MDS
2. client try I/Os to a device he doesn't have permission to access
3. client send the error to the MDS and ask for re-direct
4. server will try to serve the data directly to the client
5. server doesn't have permission as it didn't check
Question: when the MDS deliver the layout to client can it be
aware that all the storage devices are accessible to the clients?

> server cannot access its storage devices.  This is not a protocol issue.
So, there is no fallback to NFSv4? My point is that there is some
uncertainty related to a layout sent by the server without knowing
that the client cannot access and it is found out at I/O time.
You are right that the NFS protocol should not be involved but is
this the correct behavior at I/O time?

> only be granted to regions of the storage device as layouts are given to
I agree that accessibility is the only thing we need to address but
if there is a fallback and the MDS cannot access data what happens.
So, what I imply is that we only need to have the right error
logging mechanism in place and detect the access problems at mount time
not at I/O time if we can.

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 18 Jun 2009 11:39:31 -0700

> aware that all the storage devices are accessible to the clients?

The NFSv4.1 protocol allows you to serve the data through pNFS, or
through reads and writes through the MDS. If neither is available, then
you are out of options as far as the protocol is concerned: just return
NFS4ERR_SERVERFAULT, and autodial the maintenance folks...

Trond

---

From: "Muntz, Daniel" <[email-masked]>
Date: Thu, 18 Jun 2009 12:07:42 -0700

> -----Original Message-----
> From: sfaibish [mailto:[email-masked]] 
> Sent: Thursday, June 18, 2009 7:26 AM
> To: nfsv4 list
> Subject: [nfsv4] Access permission issues in pNFS
> 
> 
> permission to access a storage device. From the pNFS protocol 
> perspective there is no error mechanism to inform that a 
> client doesn't have the access permission to a storage device 
> at mount time. This is also the case with the MDS that 
> doesn't have access permission to some storage devices.
> 
> LUNs but the client doesn't have the access permission to the 
> LUN the client will send an error to the server and it will 
> mount as NFSv4.1 without pNFS support (fallback to NFS) and 
> the error is logged both at the client and the server.
> storage devices it will log an error at the client and at the 
> server. In this way an admin can take corrective measures on 
> either server or client or both.
> 
> serve the I/O to the client still without logging an error at 
> least not at mount time. In the usecase when a new storage 
> access permission to the new storage device from this time on 
> all the IOs to the new storage device will be served by the 
> MD server creating a performance and scalability bottleneck 
> without possibility of detection.
> 
> appropriate to be recommended as an implementation option on 
> the client.
> 
> unhealthy pNFS server.

An MDS will routinely access data stores on behalf of clients that don't
speak NFSv4.1.  The I/O-through-MDS case is much less likely to be an
error path than to simply occur as part of normal operation for v2/3/4.0
clients.

> Additional if we agree on an error 
> mechanism it might be better to be used at mount time than at 
> I/O time and perhaps to be logged by the server and the 
> client in either case.

Doing this at mount time is an optimization, because you must still do
it for I/O.  A notification mechanism would be nice to have so layouts
could be created with available replicas.  Without such a mechanism, the
client should fall back to I/O through the MDS.

How often will we avoid falling back to I/O through the MDS if this is
implemented (i.e., server has replicas that are available to the
client)?

> Missing piece of protocol: Client has no obvious way to 
> report an inaccessible device to the server.
> 

Is this proposal suggesting that the client should do I/O through the
MDS for inaccessible portions of a file, but use pNFS for accessible
portions?  E.g., should there be something in the layout to indicate
that for certain ranges the client should do I/O through the MDS.

> ellaborate on this). The intention should be not to change 
> 4.1 (no doc respin) and to wait for 4.2 and add new method.
> 
> Your thoughts?
> 
> /Sorin

---

From: sfaibish <[email-masked]>
Date: Thu, 18 Jun 2009 15:46:00 -0400

> error path than to simply occur as part of normal operation for v2/3/4.0
My reading of the protocol is that the MDS in a pNFS only server/client
setup will not test the access to the data stores before the I/O
time. So, the MDS will only find that it has no permission access to
some data stores so the client I/O will fail both the forward and fallback
cases but I am not sure what error will be logged. Additional to this
let's say we accept this situation and all the clients that have permission
deny to a certain data store will do all the I/Os via MDS which will be a
scalability problem.

> client should fall back to I/O through the MDS.
I totaly disagree. It is not an optimization but reducing the uncertainty
in the system and enabling the detection and resolve of the problem before
the clients start to access the storage during a large job that can be
detected perhaps long inside the job like the last day before a 3 week HPC
run. This will result in 3 weeks of wasted time and resources. And this
is not related to the layout and replicas it is related to the fact that
there is uncertainty in the system. Also in some cases an admin might
want to remove the data store until fixing the problem and not stop
all the I/Os of clients or do them via the MDS. It is a gap in the  
protocol.

> implemented (i.e., server has replicas that are available to the
It is not how often but at which point in the data access: mount time
or I/O time. It will prevent running jobs that will fail on the last I/O
even if it has a valid layout from the MDS. It might be possible for
the server to have replicas for the client but this is more the exception
than the rule and it is outside of the protocol.

> that for certain ranges the client should do I/O through the MDS.
We are trying to address gracefully this situation, I mean as graceful
as we can. I would very much want to prevent this situation ever or
at least not so late in the data access. You have but a valid point
that it might be good to communicate to the client for each layout
member the I/O path  but it will be an overkill in my opinion.
Also the scalability problem will remain in place if you apriori
decide that I/Os will be served by the MDS. I would think that
it is better to fix the problem before the run. On the other hand
you could address cases when intentionally the admin doesn't want
a certain client to access a data store directly.

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: "Halevy, Benny" <[email-masked]>
Date: Thu, 18 Jun 2009 16:53:30 -0400

> time. So, the MDS will only find that it has no permission access to

Would checking this a getdeviceinfo time help?

> deny to a certain data store will do all the I/Os via MDS which will be a

But what other choice is there?
If you want to use pnfs you need to make sure the client
can access storage directly.  If it can access only part of it
then you need to configure the MDS so it knows which layouts can
be handed out to which clients.  I don't think this problem
should be solved via the nfs protocol.  It seems like  a system
configuration issue specific to the block layout type probably
over a specific storage fabric (e.g. FibreChannel)

Benny

-----Original Message-----
From: [email-masked] on behalf of sfaibish
Sent: Thu 2009-06-18 12:46
To: Muntz, Daniel; nfsv4 list
Subject: Re: [nfsv4] Access permission issues in pNFS

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: sfaibish <[email-masked]>
Date: Thu, 18 Jun 2009 17:21:14 -0400

> Would checking this a getdeviceinfo time help?
It will be a step in the right direction. So, at mount time or immediately  
after
the client will issue a getdeviceinfo but how does the server know that  
information.
Still the client need to check somehow permission to data stores. Should  
we make
this an implementation decision?

> can access storage directly.  If it can access only part of it
Or to check and if not to log an error. I am for the error logging more
than for fixing that must be done by an admin outside the protocol.
And to detect the problem before the I/O phase.

> be handed out to which clients.  I don't think this problem
I agree but I thought Dan's idea was worth exploring. Also I
am mostly looking at the case when an entire store cannot be accessed
rather than parts of it.

> over a specific storage fabric (e.g. FibreChannel)
Remember the block is all set on this as I mentioned in my first email.
If the server cannot access a device it cannot mount the FS and as such
the client cannot mount the pNFS at all and error is logged. If the client
cannot access a lun the mount with pNFS support will fail and again
an error is logged both at client and server. The problem is with
file and maybe OSD layouts when there is no mount time check.

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: "Halevy, Benny" <[email-masked]>
Date: Thu, 18 Jun 2009 17:34:49 -0400

> the client will issue a getdeviceinfo but how does the server know that  

Good question.  But out of scope for NFS.

> we make this an implementation decision?

Absolutely.
It seems to me that if your system is configured properly,
including the SAN and the MDS, then the MDS can make the
device list per-client so it includes only devices it's
supposed to have access to and if the layout refers
to devices that the client is not supposed to be able
to access then the MDS should return NFS4ERR_LAYOUTUNAVAILABLE
to this client's LAYOUTGET.

In this case, if the client is denied access to storage
it should assume that something went wrong causing it
to be fenced off.  I think that recovering from that
would require an orderly restart of the client and
reconfiguration of the fabric.

Benny

-----Original Message-----
From: sfaibish [mailto:[email-masked]]
Sent: Thu 2009-06-18 14:21
To: Halevy, Benny; Muntz, Daniel; nfsv4 list
Subject: Re: [nfsv4] Access permission issues in pNFS

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: sfaibish <[email-masked]>
Date: Thu, 18 Jun 2009 17:52:08 -0400

> Good question.  But out of scope for NFS.
I disagree, because if you don't you may create a scalability issue
which is the oposite than what pNFS intention was. Maybe outside of NFS
but not outside of pNFS. Again, the block has not this problem as it
is reinforced by the server checking its access to devices and will not
allow the client to mount if it doesn't.

>> we make this an implementation decision?
Not sure you can do this. The protocol needs to provision for error cases
mainly for the reason of misconduct of admins that I explained in the  
previous
email.

> to this client's LAYOUTGET.
Again you refer to block layout. For file this is a problem if the server
doesn't know that a client cannot access a data store and yet it sends a
valid layout which the client cannot access and the server doesn't know.
If it would know it will perhaps send a layout not including that device.
Moreover there are 2 cases here; one when the admin doesn't want
a client to access a data store; second when the client was not
configured properly when a new data store is added (see my usecase).
In the first case wouldn't be nice if the MDS will be aware of the
situation and refrain from sending a layout using the misconfigured
data store or informing the client that for the part of the layout
on the bad data store to access via the MDS? Do you see this outside
of the scope of the pNFS protocol?

> reconfiguration of the fabric.
This is a lot of work. All that I asked is to have the right error
reporting mechanism early enough in the game to prevent impact
on performance and scalability of pNFS.

/Sorin

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: "Muntz, Daniel" <[email-masked]>
Date: Thu, 18 Jun 2009 16:01:53 -0700

> Sent: Thursday, June 18, 2009 12:46 PM

Okay, taking the assumption of a pNFS-only environment (all clients are using NFSv4.1+pNFS exclusively)...

With the existing protocol, a client _could_ verify connectivity to all DSs and fail the mount (based on a mount option) if there are not sufficient DSs to perform all I/O directly to DSs (accouting for replicas).  Alternatively, the client _could_ be implemented to allow the mount to succeed and do I/O through the MDS for the ranges where the DSs are unavailable, but use pNFS where DSs are available.

If the client I/O fails both directly to a DS and through an MDS, then you're DOA.  This would be nice to know at mount time :-)

If a particular DS has problems w.r.t. clients only, then they have to fall back to I/O through the MDS or they fail.  It's a choice between slow I/O and no I/O.

> than the rule and it is outside of the protocol.

It is more likely that failures will occur over time, possibly long after the mount, rather than being an existing condition at mount time.  Because you have to be able to handle the former, the latter, while nice if you can handle it, is an optimization.  Another thing to consider: this seems somewhat analogous to the implementation decision of whether to establish DS connections from the client at OPEN time or at I/O time.  You could do a "DS ping" during mount and/or during OPEN (you're implicitly doing it during I/O :-).

> a certain client to access a data store directly.

My point, here and above, is that the DSs may be accessible at mount time and fail later, so you still have to deal with the case where a DS becomes unavailable.

Failing a mount because a DS is not accessible is going to be very ungraceful in many cases.  Allowing a mechanism where I/O through the MDS is designated for inaccessible portions of a file is (imho) more graceful, but a burden on the client implementation (still, potentially useful).