Subject: [nfsv4] Additional discussion on Permission Access Check

From: sfaibish <[email-masked]>
Date: Tue, 15 Mar 2011 17:20:15 -0400

Per our previous discussion these are some open issues that we want to
address in the Permission Access draft that will be included in the 4.2.

1. Do we want to return multiple device errors via the FILE type?
    The current draft envisions a layoutreturn per device with an offset
    range that corresponds to that device.  Both the FSID and ALL return
    types take a device array.  My opinion is that we should try to add
    a device array to FILE as well.

2. Should we introduce/use an additional error type if there is a
    DS connectivity or inaccessible issue other than permissions? We can
    use something like NFS4ERR_CONN although 5661 recommends to
    use NXIO (or not to) for IO errors to treat connectivity/accessibility
    errors not specific to DS. We may want to add a special error case
    specific to connectivity in the LAYOUTRETURN and sending the I/O
    to the MDS.
    The problem we have is to distinguishing connectivity errors from other
    device I/O errors, as the current version of the draft uses
    NFS4ERR_IO for both. The current plan is resurrect NFS4ERR_NXIO to
    distinguish connection errors (unreachable) from other I/O errors
    that (continue to) use NFS4ERR_IO.

3. Should we add SCSI errors to DS reports for block as connectivity error?
    We include the SCSI errors in the syslog of the client but do we want
    also to report the error in the layoutreturn for the MDS?

4. In the current draft nfsstat4 will be able to report different types of
    access denial for each deviceid that is returned. Shouldn't we use  
errors
    instead of nfsstat4?

5. There will be TCP level disconnects to the DS that will be detected
    and logged by the client. Should it be enough to detect a link down
    on the DS. The result will be the same as in the case of fallback of
    large number of clients to MDS after trying all the configured paths
    and no one will know the reason. I believe this will make easier to
    detect I/O errors and not sure that NXIO is good enough as it is used.

    In fact in an indirect way 5661 forbids/discourage use of NXIO in the
    server. So assuming hat as a baseline (i.e., NFSv4.1) the net is that
    NXIO is restricted to our use case - NXIO would cover any and all
    reasons for not being able to reach the device, including TCP
    connection failure to the DS.

"15.1.16.3. NFS4ERR_NXIO (Error Code 5)

     I/O error.  No such device or address.  This error is for errors
     involving block and character device access, but because NFSv4.1 is
     not a device-access protocol, this error is not applicable."

    The spec does not actually forbids it, but it just says it's not
    applicable for the server. We want to make it applicable to the server
    as well. And indeed it is not listed as a valid error code for any
    operation, but it's still an allowable value defined in the XDR.
    As we add new operations or conditions for which it may apply we
    will use NFS4ERR_NXIO for connectivity to DS type of IO errors.

6. In the current draft we do not address explicitly the cases when the
    MDS cannot access a SD for I/Os and this will result in the
    impossibility to execute the I/O on behalf of a client that fallback
    to the MDS and result in an I/O error without a visible reason, again
    a silent error. We assumed that in this case an admin can trace this
    in the server log and fix it but there is no explicit error for this
    case neither. We tried to address this issue by making assumed access
    of MDS to DS an explicit requirement.

Please take a moment and take a look and comment on these issues. I
would like to filter and only discuss the issues that we have disagreement
in Prague. Thanks all

/Sorin

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Unified Storage Division
         EMC²
where information lives

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 16 Mar 2011 08:54:48 +0200

>     a device array to FILE as well.

For objects We have this in v4.1 already. Perhaps it can be crafted with a
united generic structures.

>     that (continue to) use NFS4ERR_IO.

I suggest to use the objects (4.1) enum for this, it has all these sates and
more. Which has been proven in the field.

>     also to report the error in the layoutreturn for the MDS?

In objects I have a pretty good conversions from scsi-errors to 
objects-layoutreturn errors. But an additional info field will not be bad
as well.

>     of MDS to DS an explicit requirement.

I'm not against an explicit error, but please note that it will not alter the
clients behaviour. In hard mounts it will get stuck for ever until connectivity
is restored. and in soft mounts it will return to the application. For that case
perhaps it is good to return the correct error code, to be printed to the user.

> would like to filter and only discuss the issues that we have disagreement

objects has a robust error returns please craft the generic returns similar
to what objects have today?

> /Sorin

Thanks
Boaz

---

From: sfaibish <[email-masked]>
Date: Wed, 16 Mar 2011 14:37:33 -0400

> more. Which has been proven in the field.
We are in fact using some of the xdr from 5664 but we can use more. I
would like Benny's opinion on this. I have all the intention to use
any good ideas from 5663 and 5664 as long as they can be made suitable
for any generic layout. We can discuss details in Prague.

> objects-layoutreturn errors. But an additional info field will not be bad
I was of the same opinion but David Black was not convinced although
he proposed such an idea first.

> perhaps it is good to return the correct error code, to be printed to  
Not sure I agree. Of course the client will get an I/O error but
the value of such "explicit" will make the error non-silent such that
an admin can take correction actions in the case that both clients and
the MDS cannot perform I/O to a DS.

> to what objects have today?
As I mentioned before we were inspired by 5664 in many cases in the draft.
So, I agree that we need to bring the other layouts to same level as
object where possible. This will actually unify all the layout types
behavior for such type of errors. Thnaks

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 18 Mar 2011 03:40:44 -0500

>   a device array to FILE as well.

Isn't the device array implicitly presented in lrf_body?

>   that (continue to) use NFS4ERR_IO.

There are two type of errors in the proposal:

]   o  If the client is unable to access the storage device, NFS4ERR_NXIO
]      SHOULD be used.
]
]   o  If the client is able to access the storage device, but permission
]      is denied, NFS4ERR_ACCESS SHOULD be used.

If we want to distinguish NFS4ERR_IO because of I/O errors in general
versus connectivity issues, then we can introduce a new error code
that only has this meaning.

We might want to do this anyway if either:

1) The blocks layout can return NFS4ERR_NXIO for any other reason.

2) We can envision a scenario where with file layouts, NFS4ERR_NXIO
might be valid. I.e., the way this is worded:

]   o  For the pNFS file layout, an indicative NFS error from a failed
]      read or write operation on the inaccessible device SHOULD be used.

could be read to pass up an error code 5 from a lower level to the NFS
layer. And that would go on the wire as NFS4ERR_NXIO.

>   also to report the error in the layoutreturn for the MDS?

Why do we care about disk level IO errors being conveyed to the MDS? These
may be transitory.

This would be NFS4ERR_IO. And the MDS can chose to respond to enough
NFS4ERR_IO errors from a specific DS by not using it in a layout.

>   access denial for each deviceid that is returned. Shouldn't we use errors
>   instead of nfsstat4?

nfsstat4 is what is used to convey the error codes.

Look at the examples such as 18.44.2:

18.44.2.  RESULT

   union layoutreturn_stateid switch (bool lrs_present) {
   case TRUE:
           stateid4                lrs_stateid;
   case FALSE:
           void;
   };

   union LAYOUTRETURN4res switch (nfsstat4 lorr_status) {
   case NFS4_OK:
           layoutreturn_stateid    lorr_stateid;
   default:
           void;
   };

>   connection failure to the DS.

Again, there are two cases we care about and two error codes.

>   will use NFS4ERR_NXIO for connectivity to DS type of IO errors.

Again, either reuse it or create a new error code.

>   of MDS to DS an explicit requirement.

Which is it? SD or DS?

The NFSv4.1 nomenclature is DS, we need to avoid adding another
confusing term.

To the main point, we need to examine what happens in a non-pNFS
model. A fibre channel cable is disconnected to a shelf  and that is logged
on the server. Does the server return specific error information to the client?
No, it returns EIO.

In other words, the error is not silent, it tells the client that the I/O failed
and the client does not need to know the specific error code.

And actually, the client may not know that the MDS is doing asynch I/O
for it. I.e., it may have never gotten a layout from the MDS and is issuing
vanilla 4.1 compounds to it.

I don't think there is any value add to making this statement explicit. I
also think the following:

]  For the specific case of adding a new storage device to a 
]  filesystem, MDS issuance of test I/Os to the newly added device 
]  before using it in layouts avoids this problem scenario, but does not 
]  cover loss of access to existing storage devices at a later time. 

both states an implementation assumption and shows the general
problem. The MDS performing I/O to the DS is certainly implementation
specific (in a cluster, the MDS and the DS may be the same device).

And if the DS becomes inaccessible to the MDS later on, that gets
reflected back to the client with an I/O error.

I'd like to add another point of discussion:

>   An important example is addition of
>   a new storage device to which a large population of pNFS clients
>   (e.g., 1000s) lacks access permission. Layouts granted that use this
>   new device, result in client errors, requiring that all I/Os to that
>   new storage device be served by the MDS server. This creates a
>   performance and scalability bottleneck that may be difficult to
>   detect based on I/O behavior because the other storage devices are
>   functioning correctly.
>
>   The preferable approach to this scenario is to report the access
>   failures before any client attempts to issue any I/Os that can only
>   be serviced by the MDS server.  This makes the problem explicit,
>   rather than forcing the MDS, or a system administrator, to diagnose
>   the performance problem caused by client I/O using NFS instead of
>   pNFS. There are limits to this approach because complex mount
>   structures may prevent a client from detecting this situation at
>   mount time, but at a minimum, access problems involving the root of
>   the mount structure can be detected.
>
>   The most suitable time for the client to report inability to access a
>   storage device is at mount time, but this is not always possible.
>   If the application uses a special tag or a switch to the mount
>   command (e.g., -pnfs) and syscall to declare its intention to use
>   pNFS, at the client, the client can check for both pNFS support and
>   device accessibility.

As a point, both the NFSv4.0 and NFSv4.1 specs go to great length to NOT talk about
MOUNT and options.

But remind me how a DS gets into this predicament? I.e., why does a DS
necessarily care about whether a client has access rights to it or not? I view
that as metadata type of information.

The client can either access that part of the namespace or it can not.
If the MDS allows a mount, then the client is eventually given a
stateid that it passes into READ or WRITE.

When the DS gets a COMPOUND with such a stateid from a client it
has never seen, it has to consult with the MDS to determine if

a) The stateid is valid.
b) The client has appropriate read or write permissions to that part of
the name space.

A key point here is that the DS may have no notion of where in the namespace
this data resides. As it may have no knowledge of the name space, exports
may not make sense. I.e., an export has two unique keys, the exported path
and the file handle/fsid of the root of that path.

If the MDS implements per-NFS operation access checks (to catch a change
in either an export policy or a netgroup), the export is still on the MDS and
not on the DS. If the DS is asynchronous, then there should exist some
heartbeat mechanism and a way for the DS to check to see if access cache
values have changed. If the MDS and the DS suffer a network partition,
then it is a server implementation issue as to whether I/O to the DS can
proceed.

I.e., I'm struggling to see why access permissions are an issue for
file layouts. I understand the problem with block layouts, but I don't
see it for files.

I do understand the utility of telling the server we voluntarily returned the
layout because of some I/O error.

What I'm getting at is that I don't buy into the use case presented.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 18 Mar 2011 12:57:02 -0500

Another way to view this proposal is to boil it down to the basics:

Client gets a layout from the server which has a set of DSes it
needs to use for IO.

Client encounters either:

1) Can communicate, but is denied permission (NFS4ERR_ACCESS)

2) Non-transient failure to communicate (NFS4ERR_NXIO)

The client wants to communicate this back to the MDS and
LAYOUTRETURN is the available means.

It needs to communicate back the following:

1) Which layout(s) it is returning:
a) Three variants - LAYOUTRETURN4_[FILE | FSID | ALL]_NO_ACCESS

2) Which DSes it could not communicate with.
a) And which error for each DS, i.e., DS1 might be NFS4ERR_ACCESS
and DS3 might be NFS4ERR_NXIO.

So yes, your point 1 is correct, you need a device array.

My counter-proposal would be to have this XDR:

   /* Constants used for new LAYOUTRETURN and CB_LAYOUTRECALL */
   const LAYOUT4_RET_REC_FILE      = 1;
   const LAYOUT4_RET_REC_FSID      = 2;
   const LAYOUT4_RET_REC_ALL       = 3;
   const LAYOUT4_RET_REC_FILE_NO_ACCESS    = 4;
   const LAYOUT4_RET_REC_FSID_NO_ACESSS    = 5;
   const LAYOUT4_RET_REC_ALL_NO_ACCESS     = 6;

   enum layoutreturn_type4 {
        LAYOUTRETURN4_FILE = LAYOUT4_RET_REC_FILE,
        LAYOUTRETURN4_FSID = LAYOUT4_RET_REC_FSID,
        LAYOUTRETURN4_ALL  = LAYOUT4_RET_REC_ALL,
        LAYOUTRETURN4_FILE_NO_ACCESS = LAYOUT4_RET_REC_FILE_NO_ACCESS,
        LAYOUTRETURN4_FSID_NO_ACCESS = LAYOUT4_RET_REC_FSID_NO_ACCESS,
        LAYOUTRETURN4_ALL_NO_ACCESS  = LAYOUT4_RET_REC_ALL_NO_ACCESS
   };

   struct layoutreturn_file4 {
         offset4         lrf_offset;
         length4         lrf_length;
         stateid4        lrf_stateid;
         /* layouttype4 specific data */
         opaque          lrf_body<>;
   };

   struct layoutreturn_device_no_access4 {
         deviceid4     lrdna_deviceid;
         nfsstat4      lrdna_status;
   };

   struct layoutreturn_file_no_access4 {
         layoutreturn_file4             lrfna_layout;
         layoutreturn_device_no_access4 lrfna_device<>;
   };

   union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
         case LAYOUTRETURN4_FILE:
                 layoutreturn_file4             lr_layout;
         case LAYOUTRETURN4_FILE_NO_ACCESS:
                 layoutreturn_file_no_access4   lr_layout_na;
         case LAYOUTRETURN4_FSID_NO_ACCESS:
         case LAYOUTRETURN4_ALL_NO_ACCESS:
                 layoutreturn_device_no_access4      lr_device<>;
         default:
                 void;
   };

That way you reuse the existing data structures and capture the
same information in all 3 cases.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 18 Mar 2011 14:14:04 -0400

> 1) Can communicate, but is denied permission (NFS4ERR_ACCESS)

Why is this needed at all? The DS can communicate this info to the MDS
via the back end protocol.

In any case, this is either signalling a file permissions change on the
MDS or, possibly a severely schizophrenic system
(NFS4ERR_SEEK_THERAPY?).

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 18 Mar 2011 13:22:06 -0500

> via the back end protocol.

Yes, which was my point last night.

Benny has described an use case for blocks - perhaps he can share
it in his own words.

> MDS or, possibly a severely schizophrenic system

And again, the way the DS knows about the permissions change
(or export policy change) is via back end protocol communication.

So it should not be a surprise for the MDS to get an error back
from the client.

And, if the MDS is smart, when either the permission changed
or the export policy changed, it should have recalled the layout.

---

From: sfaibish <[email-masked]>
Date: Fri, 18 Mar 2011 14:23:26 -0400

I was just answering your previous email when this came. I think this is
very good proposal and addresses some of my concerns. It is in line with
my suggestions. I still need to asnwer your questions from the previous  
email
including if this is a valid usecase, which in my opinion is and Mike  
agreed
to it during our long discussions in Hiroshima. Thank you for taking the
action and make a proposition. I would still like to hear David Blacks
and Jason Glasgow opinion.

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: sfaibish <[email-masked]>
Date: Fri, 18 Mar 2011 16:11:28 -0400

> Isn't the device array implicitly presented in lrf_body?
If you look at the structure it is only addressing one layout segment.
This is exactly what I wanted to be extended to a list of segments
as it is the case for FSID and ALL. If there are multiple segments
to multiple devices with permission access you will need to send
multiple layoutreturns. Of course it can be done in one COMPOUND but
it make sense to me to send the list of layouts that use the DS with
the access issue in one layoutreturn.

> that only has this meaning.
This was exactly my question, should we use a different error code
for connectivity issues which could be temporary while permission
denial is permanent unless admin intervention.

> 1) The blocks layout can return NFS4ERR_NXIO for any other reason.
Again, my concern/focus of the draft is file layout not block. for block
the only question was if to propagate the SCSI errors to the MDS or
just in the client syslog. The connectivity is mostly related to network
not to SCSI; SCSI connectivity result always in SCSI errors as the
block client doesn't know that the cable was yanked or that
the admin removed the access to storage group.

> layer. And that would go on the wire as NFS4ERR_NXIO.
Well, the question here is should the client inform the MDS that he
cannot connect to the DS included in the layout as the reason for
sending the I/O to the MDS. Of course I assume that the client will
differentiate between connectivity errors and normal I/O errors.
Of course it will always fallback to the MDS but if there is a real
connectivity issue to access the DS and if it persist the
MDs should not include that device in the layouts it sends to the
client. I just want the MDS to have the information. And there is one
more little detail; the MDS may send a layoutrecall when it receives
the I/O request from the client that has a valid layout for that DS.
The error and layoutreturn will prevent the need for the CB.

> Why do we care about disk level IO errors being conveyed to the MDS?  
This is true but what I propose is to report the SCSI errors only after
max number of retries expired, or as David Black explained, when the client
is "disgusted" with the errors.

> NFS4ERR_IO errors from a specific DS by not using it in a layout.
Yes but my question was, is this enough to allow an admin to take
correction actions from the syslog? Plus do we want the MDS to keep
track of all the erorors until deciding to remove the DS from the
layout? Besides is this a client specific or global issue including
that MDS might have a similar problem when sending I/Os to the DS.

> nfsstat4 is what is used to convey the error codes.
OK. I was just wondering if status can be sent on MDS request while errors
will be sent by the client. (just semantic for my understanding).

> Again, there are two cases we care about and two error codes.
If you include connectivity in NXIO I agree but the server side.
The client can use NXIO but without specifying the error code/reason
it will be less useful for an admin.

> Again, either reuse it or create a new error code.
I suggested to use a new error code as it is not so expenssive if the
client can distinguished between errors. Otherwise we can add an error
that will be never used. Better not add one like this

>>   of MDS to DS an explicit requirement.
DS, simply the diagram is using SD in error. It should be DS in all places.
Need to fix this.

> on the server. Does the server return specific error information to the  
This is not a problem as the client will always fallback to MDS which
cannot mount the FS if the FC cable is disconnected. I am not sure I get
your point here.

> and the client does not need to know the specific error code.
You mean the SCSI errors? Why should the non-pNFS client worry about FC
connectivity? It will be the problem of the MDS and the client will never
know except it will receive and EIO from the MDS server and will continue
to retry, possibly forever.

> vanilla 4.1 compounds to it.
Don't understand what you are saying here. If there is no layout for
the faulty DS there is no permission problem related to this very draft.
It is solely the problem of the MDS not a protocol issue for both client  
and
server.

> specific (in a cluster, the MDS and the DS may be the same device).
Maybe but we need to address the issue in all the possible MDS
implementations as we do not mandate a cluster and even if we do
there is still file striping to be used when the DS can have
permission issues for clients.

> reflected back to the client with an I/O error.
I agree, but my question was if we should log the error in the MDS
and it was already reported by clients before more clients try and
get stuck and the whole system hangs on EIO without the knowledge why
and how an admin can correct the situation.

> that as metadata type of information.
As I mentioned before there is a fencing mechanism to fence the client  
which
is based on permission access. Same mecanism can be used to explain how
a client has no permission to access a DS by a configuration mistake
or by wrongly fencing the DS and if the client doesn't report this  
condition
there is no way to know that there is a fencing in place for example on an  
MDS
crash situation. Could be other examples and usecases.

> stateid that it passes into READ or WRITE.
Why do you make it a namespace issue? If the MDS sent a layout
that include a faulty DS to the client it is already wrong. Do
you propose for the MDS to check the access to all the DS's that
it includes in layouts? I am not against but I think what we
propose here is lighter/simpler and less demanding from the MDS.

> b) The client has appropriate read or write permissions to that part of
This is not a namespace question or I don't understand your point.
It is a layout validity question in my mind; a layout including faulty
devices is illegal and we would like at least to know about this so
the MDS or admin take corrective actions at a later time to allow
preventing additinal clients to get illegal layouts.

> file layouts. I understand the problem with block layouts, but I don't
It is the don't know how many times I tell you; this is mostly a file
layout issue more than block or object. As I explained the block and
object are more strict on permission access reinforcement. It can happen
but it is less of an issue.

> layout because of some I/O error.
OK, this is good because 5661 doesn't protect explicitly against
illegal/invalid layout sent to clients.

> What I'm getting at is that I don't buy into the use case presented.
After this email I hope I explained better. But I would like to discuss
all these issues in Prague please. Thanks for taking the time.

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 18 Mar 2011 16:26:24 -0500

On Mar 18, 2011, at 3:11 PM, sfaibish wrote:

> On Fri, 18 Mar 2011 04:40:44 -0400, Thomas Haynes <[email-masked]> wrote:
> 
>> 

The text introduces the term 'storage device.'

> cannot mount the FS if the FC cable is disconnected. I am not sure I get

Ahh, I was too terse.

In NFSv4.0, if the server has problems talking to a disk shelf, does that
error message get sent back to the client?

My point is that you have to always consider what a non-pNFS system
will do and what the equivalent cases are here.

> It is solely the problem of the MDS not a protocol issue for both client and
> server.
> 
>> 
> As I mentioned before there is a fencing mechanism to fence the client which
> or by wrongly fencing the DS and if the client doesn't report this condition
> there is no way to know that there is a fencing in place for example on an MDS

But the fencing mechanism relies on the MDS communicating to the DS. When
the MDS is informed of the fence, it can proactively inform both the DS and
any existing layout of the change.

Also, if I remember my fencing correctly, the MDS has to wait for any existing
I/O to drain. In this case, I would consider a contradictory outstanding layout
to be an existing I/O and wait to respond to the fence request only when all
layouts were recalled.

PS: I don't think it makes sense for the fence to apply to one of the DSes and
not the rest.

> Why do you make it a namespace issue?

Because export policies deal with namespaces and not files.

To have permission to access a file, you first need permission to
traverse the namespace to that get (or create) that filehandle.

That permission check comes from the export policies on the
MDS. When does the DS see a LOOKUP or GETFH?

So the client has navigated the namespace to get to a file. It
then wants to OPEN the file. It issues such a compound to the
MDS. And the MDS then checks to see if the user has file level
permissions to access the file. If they do, then they get a file handle
back.

In the case of pNFS, they also get a layout.

The client now wants to do a READ or a WRITE to the file. It issues
a compound to the DS.

If the DS wants to check some export policy, how would it do so?

It has no notion of where the file resides in the namespace. Granted
it could look at the FH and determine that this I/O was occurring
in a pNFS data storage unit, but I still disagree that a DS should
be applying exports at such a level.

If we have 1 MDS and 100 DSes, do we want to be micro-managing
the exports on 101 machines?

> If the MDS sent a layout

That is not a permissions issue, that is an I/O error.

It is perfectly legal for a MDS to send the client a layout with a DS
which is in the process of rebooting.

But say that the DS sits in maintenance mode. So now both the MDS
and client are aware of the issue. The more interesting case is when
there is a network partition on the frontbone. I.e., the client sees it,
but the MDS does not.

And in that case, this is the connectivity issue and not a permissions
issue.

And I have agreed that it is nice for the client to be able to report
something it can detect and the MDS can not detect.

> devices is illegal

It is not illegal, sub-optimal yes, illegal, no.

The argument here is mixing the two issues:

1) Permissions - the client can talk to the DS, but is not granted access.
In this case, the DS can report that fact to the MDS. If there is a backbone
network partition, then the MDS is aware of the issue already. (This is
implementation specific, but a rule could be if the DS looses connection
to the MDS, then existing stateids keep the same access and new
stateids are denied access. But the MDS knows that.)

2) Connectivity - the client can not talk to the DS at all. In this case,
the MDS may or may not be aware of the issue. In NFSv4.1, the
client has two options:

a) Wait until the connectivity heals itself.

b) Ask the MDS to channel the I/O.

> and we would like at least to know about this so

WIth permissions, the MDS should not be issuing such a layout. And
if the file level permission or export policy changes, the prudent MDS
should be recalling the outstanding layouts.

With connectivity, yes, it would be nice for the MDS to know not to issue
such a layout for some time. 

> all these issues in Prague please. Thanks for taking the time.

Please do not mix up my still not buying the use case with not
buying the need for communicating connectivity issues back to the
MDS.

I'm fine with taking the modified XDR I gave earlier. I am not fine
with the offered explanation of how exports work on DSes 
and/or fencing.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 18 Mar 2011 20:54:58 -0400

> Benny has described an use case for blocks - perhaps he can share

Please do.

> or the export policy changed, it should have recalled the layout.

Exactly...

> >> 2) Non-transient failure to communicate (NFS4ERR_NXIO)

So this is what I still don't understand. Remote syslog is a problem
that has already been solved. Why do we have to solve it again with yet
another NFS extension?

What exactly is the problem that people are trying to solve here and
why, as a client developer, should I give a damn?

NetApp

---

From: Benny Halevy <[email-masked]>
Date: Sun, 20 Mar 2011 09:55:47 +0200

> for any generic layout. We can discuss details in Prague.

We discussed this in Maastricht and decided that we'll keep both methods
rather than deprecating RFC5664's method in favor of this one.
So that pnfs over objects can return detailed I/O error information
to the server to be used for graceful recovery from I/O errors during
client-RAID writes.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Sun, 20 Mar 2011 10:13:40 +0200

>> Benny has described an use case for blocks - perhaps he can share

With objects and blocks (SPC-4) the client can get the equivalent of a NFS4ERR_ACCESS
error when the capabilities it has in hand expire or get invalidated (revoked) by the server.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 20 Mar 2011 12:36:09 -0400

> error when the capabilities it has in hand expire or get invalidated (revoked) by the server.

But the server knows when they expire, and can revoke the layout. Why
does it need to be told by the client?

NetApp

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 20 Mar 2011 12:37:00 -0400

> But the server knows when they expire, and can revoke the layout. Why
                                                 ^^^^^ recall

NetApp

---

From: "Mike Eisler" <[email-masked]>
Date: Sun, 20 Mar 2011 14:14:53 -0700

> why, as a client developer, should I give a damn?

Customers will have a better experience if pNFS is used to send an I/O to
a DS, than if non-pNFS used to send an MDS. If there is not rough
consensus on this point then I want the last years of my life back and
will start collecting from attendees at the Prague meeting next week.

Storage clusters are more complex than monolithic storage arrays. Thus
things might more easily break depending on the design of the product or
the deployment.

Storage administrators, network administrators, compute server
administrators, and application administrators often don't talk among
themselves. In fact, in many cases they loathe each other.

When there is a problem, there is finger pointing. No one wins, and only
the end-user of the application  loses.

The proposal David Black, Sorin, and I have made is to provide more
information about why pNFS has stopped working. Yes, I would expect a
syslog from the storage cluster, but an a distinct error returned to the
client ought to also generate a syslog and might cause the compute server
admin or application admin to get in the face of the storage admin sooner
and provide an end-user relief sooner.

Backing off to the MDS is not the solution. Once the end-user gets used to
lower latency and higher throughput of I/O direct to DS, he will be
unhappy when it goes away.

On Fri, March 18, 2011 5:54 pm, Trond Myklebust wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 20 Mar 2011 17:42:19 -0400

> and provide an end-user relief sooner.

In this case, we're talking about the _client_ having to report back to
the server what is wrong with said server through two different
channels: syslog _and_ nfs.

> unhappy when it goes away.

As far as I can see, the whole proposal is based around returning the
layout. Is the client then supposed to the turn around and immediately
ask for a new one?

I believe you yourself have quoted the definition of insanity in several
IETF meetings previously: trying the same thing more than once and
expecting different results. I'm having trouble seeing how a protocol
change will help us avoid insane behaviour here.

Trond

NetApp

---

From: "Mike Eisler" <[email-masked]>
Date: Sun, 20 Mar 2011 15:02:26 -0700

inline ...
> channels: syslog _and_ nfs.

yes, that's true, and doesn't detract from the point being made.
> layout. Is the client then supposed to the turn around and immediately

Most of the time immediately won't work, since in most cases, human
intervention is needed. You raise a valid point that the proposal (AFAIK)
does not provide advice for when the client should try again.

One suggestion would be to tie this to a device ID. If the device ID
doesn't work, the server needs to stop providing layouts that use it. If a
subsequent LAYOUT on the same file (or any file for that matter) uses the
same device ID, then that is an indication that the server is still messed
up. The cost of LAYOUTGET is still incurred, but we avoid the cost of an
operation to the DS.

Beyond that, adding a new callback goes beyond the scope for NFSv4.2 (IMO).

> expecting different results. I'm having trouble seeing how a protocol

I use that definition several times a day; it makes me very unpopular.

> change will help us avoid insane behaviour here.

It won't avoid it; it will provide a means to report it.

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 20 Mar 2011 21:05:54 -0500

> layout. Is the client then supposed to the turn around and immediately

And this only helps on creates - if the file exists, the client ca whine all it wants about a certain DS, but the data is there.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 21 Mar 2011 20:33:46 +0200

>> does it need to be told by the client?

This condition is not supposed to happen, hence it is an error.
Normally, the client would not use a layout with a capability that is about to
expire but rather it would return/drop it in time and ask for a new layout.
(We want to avoid recalls in the normal case)
But unexpected expiration can happen due to timeouts and retries in lower layers.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Mon, 21 Mar 2011 20:38:18 +0200

inline ...

>> channels: syslog _and_ nfs.

What about clients that do not have syslog?
Say, a database that talks (p)NFS, for example.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 21 Mar 2011 14:58:57 -0400

> Say, a database that talks (p)NFS, for example.

Why would that not have syslog? It certainly has if you are running on a
POSIX system. I imagine that Windows has similar functionality.

Even if you don't specifically use the posix syslog for logging, there
are plenty of other remote management tools out there that will allow
you to monitor arbitrary logfiles for specific events. Administering
large clusters would be utterly impossible without that...

NetApp

---

From: "Welch, Brent" <[email-masked]>
Date: Mon, 21 Mar 2011 13:55:54 -0700

Please remember that the pNFS model supports passive storage devices,
including block RAID controllers and OSD (or OST) that do not, or cannot, proactively
tell the metadata server anything.  Instead, they use the SCSI framework and they return
errors to the clients that access them.  The client relays that information to
the metadata server, either with the error information associated with layout return or
commit, or by simply returning the layout and trying again.

If you have an urge to optimize this error path, resist it.  Not only may it be
impossible given the characteristics of the storage device, but it isn't important.

Note that reliance on the back-end protocol to optimize this probably means you have
a non-optimal regular I/O path that implies chatter between the data servers and meta
data server during normal I/O operations.  The pNFS architecture  is designed so you can
eliminate that back-end chatter.  In addition, this is not "files vs blocks and objects"
because there could be files-based pNFS implementations that similarly optimize the
regular I/O path and depend on this error communication channel.

>>>Benny Halevy said:
--
Brent Welch
Director, Software Architecture, Panasas Inc
The Leader in High Performance Scale-out NAS

www.panasas.com
[email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 21 Mar 2011 17:06:34 -0400

> regular I/O path and depend on this error communication channel.

No there can't.

A pNFS files data server that returns EACCES in conditions when the MDS
would allow it is purely and simply broken.

Trond

NetApp

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 21 Mar 2011 16:52:58 -0500

On Mar 21, 2011, at 4:06 PM, Trond Myklebust wrote:

>> the metadata server, either with the error information associated with layout return

Which does not exist in NFSv4.1, which is what Sorin, et. al, are proposing.

>> or

12.2.6.  Control Protocol

   As noted in Figure 1, the control protocol is used by the exported
   file system between the metadata server and storage devices.
   Specification of such protocols is outside the scope of the NFSv4.1
   protocol.  Such control protocols would be used to control activities
   such as the allocation and deallocation of storage, the management of
   state required by the storage devices to perform client access
   control, and, depending on the storage protocol, the enforcement of
   authentication and authorization so that restrictions that would be
   enforced by the metadata server are also enforced by the storage
   device.

   A particular control protocol is not REQUIRED by NFSv4.1 but
   requirements are placed on the control protocol for maintaining
   attributes like modify time, the change attribute, and the end-of-
   file (EOF) position.  Note that if pNFS is layered over a clustered,
   parallel file system (e.g., PVFS [51]), the mechanisms that enable
   clustering and parallelism in that file system can be considered the
   control protocol.

The management of state is a big ticket item here. With a files
approach, the client is presenting both a stateid and a file handle.
The DS really needs to validate these. And that means the control
protocol.

Also, Section 12.9 talks about per-op security checks - this means
access checks against exports. Section 13.11 also mandates that
for every operation, the combination of FH and stateid be checked
for a valid layout. Which again, means the control protocol.

Section 12 and 13 suggest that access permission can be encoded
in the file handle, but from a security viewpoint, I think this is a major
PITA with AUTH_SYS.

> would allow it is purely and simply broken.

Agreed - if a fencing has occurred, then by definition the MDS would not
allow access and it should not require the client to tell the MDS that the
DS did not allow access.

---

From: sfaibish <[email-masked]>
Date: Mon, 21 Mar 2011 18:02:46 -0400

Not sure I follow what you are saying here? If it is a server or client
optimization how would they agree to match each other assuming the
different parties own the server and the client and not both. I
would like very much to understand what is your point otherwise
let's discuss how we can get the best drfat and not arguing about
the need. Thanks

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]