Subject: Re: [nfsv4] [pnfs] Summary of discussion on pNFS permission access draft

From: Benny Halevy <[email-masked]>
Date: Thu, 19 Nov 2009 17:55:57 +0200

On Nov. 19, 2009, 17:41 +0200, sfaibish <[email-masked]> wrote:
> During the last IETF meeting me and Mike Eisler discussed the proposed  
> draft
> and came to some conclusions that I would like to open for discussions to  
> the
> pNFS team. The summary might repeat itself at times or may not be in the  
> right
> order but reflect accurately our discussion.
> 
> 1. It must be a protocol change and not an implementation optimization if  
> both
>     server and client have to change their behavior.
> 
> 2. The first step is to implement an error mechanism comunicated between  
> client
>     and server already in the draft. So, please take your time and review  
> it:
>     http://www.ietf.org/internet-drafts/draft-faibish-nfsv4-pnfs-access-permissions-check-01.txt
> 
> 3. The protocol change will not address the detection of the exception  
> cases
>     that more than one client falls back to the server due to permission  
> access to
>     a DS and will be left as an optional recommendation or implementation  
> detail
>     or implementation optimization. The protocol will just define the  
> permission
>     checks to identify the problem as a permission fault to allow to inform  
> the
>     admin about the denial.
> 
> 4. We should make recommendation of when this mechanism be used even if  
> there will
>     require a specific optimization of the server it is good to bring this  
> case to the
>     attention as a recommendation optional to hep developers decide how to  
> address it.
> 
> 5. The mds will detect the situation and might do several things: can send  
> a
>     LAYOUTRETURN CB for the layout that points to the DS with permission  
> problem and
>     send a new layout for the opened file at write time; can check that  
> itself has
>     permission access to the DS with the problem and report an error  
> requesting
>     fixing the problem as the i/os on behalf of the client might fail; can  
> perform
>     the i/o on behalf of the client as a fallback to nfs; can check if the  
> client
>     has permission access to the ds; mds can go back to the client and ask  
> to perform
>     a permission check on a ds using a new layout command.
> 
> 6. There will be no treatment for the case when the mds will take care of  
> other
>     operations on behalf of client like encryption or compression and the  
> mds will
>     have to execute the i/o on behalf of the client it is outside of the  
> scope of
>     this feature. The only reference to this case will be to differentiate  
> the
>     permission access issue from the special cases described above as well  
> as any
>     fallback that will be used in the normal cases of recovery after a  
> client or
>     a server crash as intended in 4.1 or any other legitimate fallback  
> situation.
> 
> 7. A new layout operation like "LAYOUTPERMCHK" will be introduced in 4.2
> 

What does it suppose to do?

> 8. It is recommended to introduce a new layout operation for error report  
> of this
>     case in 4.2 (addressing 3.2 in the draft).

With objects layout the client can already reports I/O errors in
LAYOUTRETURN using the lrf_body layout-type specific member.
See http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12#section-7
and http://tools.ietf.org/html/draft-ietf-nfsv4-pnfs-obj-12#section-8
Can you use a similar method for the blocks layout type?

> 
> 9. We agreed that this is a scalability problem for pNFS in 4.1 and at a  
> minimum it
>     needs to be addressed by error report for the admin to take  corrective  
> actions.
>     An amendment can be done in 4.1 by extending the current object error  
> and the
>     opaque.
> 
> 10. When the permission check fails the mds will take action to send the  
> client
>      cb_layoutrecall for the write layout owned by the client with  
> permission deny
>      problem as it is expected the client to perform a layoutreturn  
> operation before
>      the fallback to nfs to prevent the cb_layoutrecall to allow the3 mds  
> to perform
>      the i/o on behalf of the client. It will be left as an optimization  
> for the mds
>      server to perform cb_layoutrecall to all the clients after detection  
> of the
>      problem and remove the failing ds from the list of valid dss until the  
> requiring
>      admin intervention to add the ds server to the mds. The protocol will  
> define this
>      usecase and recommended remedy strategy. Should we add a new layout  
> command for
>      this operation that will recall the layout and replace the ds  
> unaccessible part
>      of the layout with a different ds layout to reduce the traffic between  
> server
>      and clients in case of large number of clients.

Is this error scenario so common it's worth adding protocol
complexity for optimizing its handling?

Benny

> /Sorin

---

From: sfaibish <[email-masked]>
Date: Thu, 19 Nov 2009 11:56:48 -0500

> What does it suppose to do?
The discussion should be included in the notes of the IETF WG meeting of
Trond.
This new operation will check that the client has permission to access a
certain DS/device for which it has valid layout but the I/O failed and
will return the server the layout for that device and include an error
report of the condition. This is one alternative or it can be response to
a server CB LAYOUTRECALL. The idea is to prevent the server to send a
layoutrecall when the client fallback to NFS and somehow warn the server
of the permission access issue and allow the server to send a new layout
removing the problematic device from the layout without having to
make the CB. This is the client being pro-active.

> Can you use a similar method for the blocks layout type?
This is exactly what I propose in section 3.1 of the draft.

> complexity for optimizing its handling?
According to some discussions we had in the IETF there are additional
usecases that can be addressed as per Marc Eshel opinion. I will leave
this to Marc to comment.

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]