Subject: [nfsv4] Use of NFS clients as pNFS data servers...

From: Trond Myklebust <[email-masked]>
Date: Mon, 06 Jul 2009 19:24:48 -0400

Hi,

I've just posted a version 0 draft of a proposal for a minimal protocol
extension to allow sharing of cached read-only data among NFS clients.
The proposal intends to add a back-end protocol (a.k.a. 'control
protocol') to allow certain trusted NFS clients to register as file
layout type data servers, and to allow them to communicate with a
metadata server in order to manage layouts.

Please see

http://www3.ietf.org/proceedings/staging/draft-myklebust-nfsv4-pnfs-backend-00.txt

Cheers
  Trond

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 06 Jul 2009 19:29:50 -0400

> http://www3.ietf.org/proceedings/staging/draft-myklebust-nfsv4-pnfs-backend-00.txt

Urgh. The URL is failing. OK, see the following attachment then...

Cheers
  Trond

Network File System Version 4                               T. Myklebust
Internet-Draft                                                    NetApp
Expires: January 7, 2010                                    July 6, 2009

 Network File System (NFS) version 4 pNFS back end protocol extensions
                 draft-myklebust-nfsv4-pnfs-backend-00

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on January 7, 2010.

Myklebust                Expires January 7, 2010                [Page 1]

Internet-Draft      pNFS back end protocol extensions          July 2009

Abstract

   This document describes an extension to the NFSv4.1 draft protocol to
   allow NFS clients to act as pNFS data servers towards other NFS
   clients.

   The intention is to reduce the load on the actual data servers by
   allowing some trusted clients to share the contents of their data
   caches with other clients.

Myklebust                Expires January 7, 2010                [Page 2]

Internet-Draft      pNFS back end protocol extensions          July 2009

Keywords

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in [RFC2119].

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Description of the proposed data sharing model . . . . . . . .  5
     2.1.  NFS client acting as a pure pNFS client  . . . . . . . . .  5
     2.2.  Meta data server responsibilities  . . . . . . . . . . . .  5
     2.3.  NFS client acting as a pNFS data server  . . . . . . . . .  5
   3.  Security considerations  . . . . . . . . . . . . . . . . . . .  7
   4.  State expiration and recovery considerations . . . . . . . . .  8
   5.  Structured Data Types  . . . . . . . . . . . . . . . . . . . .  9
     5.1.  proxy_identifier4  . . . . . . . . . . . . . . . . . . . .  9
   6.  New client operations  . . . . . . . . . . . . . . . . . . . . 10
     6.1.  REGISTER_DS - Offer to act as a data server  . . . . . . . 10
       6.1.1.  ARGUMENTS  . . . . . . . . . . . . . . . . . . . . . . 10
       6.1.2.  RESULTS  . . . . . . . . . . . . . . . . . . . . . . . 10
       6.1.3.  DESCRIPTION  . . . . . . . . . . . . . . . . . . . . . 11
     6.2.  UNREGISTER_DS - Revoke offer to act as a data server . . . 11
       6.2.1.  ARGUMENTS  . . . . . . . . . . . . . . . . . . . . . . 11
       6.2.2.  RESULTS  . . . . . . . . . . . . . . . . . . . . . . . 12
       6.2.3.  DESCRIPTION  . . . . . . . . . . . . . . . . . . . . . 12
     6.3.  PROXY_OPEN - Check proxy access rights to a file . . . . . 12
       6.3.1.  ARGUMENTS  . . . . . . . . . . . . . . . . . . . . . . 12
       6.3.2.  RESULTS  . . . . . . . . . . . . . . . . . . . . . . . 12
       6.3.3.  DESCRIPTION  . . . . . . . . . . . . . . . . . . . . . 12
   7.  New callback operations  . . . . . . . . . . . . . . . . . . . 14
     7.1.  CB_PROXY_REVOKE - revoke proxy access rights to a file . . 14
       7.1.1.  ARGUMENTS  . . . . . . . . . . . . . . . . . . . . . . 14
       7.1.2.  RESULTS  . . . . . . . . . . . . . . . . . . . . . . . 14
       7.1.3.  DESCRIPTION  . . . . . . . . . . . . . . . . . . . . . 14
   8.  References . . . . . . . . . . . . . . . . . . . . . . . . . . 15
   Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 16
   Intellectual Property and Copyright Statements . . . . . . . . . . 17

Myklebust                Expires January 7, 2010                [Page 3]

Internet-Draft      pNFS back end protocol extensions          July 2009

1.  Introduction

   The object of this proposal is to allow further scale out of NFS
   traffic by allowing NFS clients to share the contents of their file
   data caches with other NFS clients.

   The model assumes a server workload in which a number of read-only
   files are commonly accessed by more than one client at a time.  A
   typical use case would be one in which the exported filesystem
   contains a set of libraries. e.g. a UNIX /lib partition, a set of
   CAD/CAM objects, or a collection of php modules and other static
   webserver data.  On such systems, a common problem occurs when
   booting up the cluster when possibly all clients need to access the
   same library data at roughly the same time.  The server bandwidth
   gets eaten up through serving up the same data over and over again.

   It is not obvious that use of the pNFS scale out mode is sufficient
   to avoid this kind of congestion.  The fundamental problem is that
   NFS clients are all accessing the same data, which are striped over
   the same data servers.  The effect may therefore simply be to move
   the bottleneck from the metadata server over onto the data servers.

   Current methods of reducing the impact of this congestion typically
   require the user to dedicate extra resources for the boot process.
   They include preloading the data on the client in local permanent
   caches (a.k.a. cachefs), replication of the shared data across
   several NFS servers and setting up NFS proxy servers.

   Another solution, which does not require the use of dedicated
   resources is the peer-to-peer model in which the first few clients to
   read the data from the server are allowed to share the contents of
   their cache with the next waves.  This RFC attempts to enable such a
   model by allowing clients which have already cached data to act as
   pNFS data servers toward their peers.  It does so by defining a
   control protocol in the sense defined in Section 12.2.6 of
   [draft-ietf-nfsv4-minorversion1-29] to enable the data server to
   enforce layouts, and negotiate authentication and authorization
   information with the server.

Myklebust                Expires January 7, 2010                [Page 4]

Internet-Draft      pNFS back end protocol extensions          July 2009

2.  Description of the proposed data sharing model

2.1.  NFS client acting as a pure pNFS client

   The proposal implies no protocol changes for NFSv4.1 clients that
   wish only to act as pNFS clients, in order to access the cached data
   from other clients.  These clients will request file layouts from the
   meta data server using the LAYOUTGET operation in the usual fashion.
   Should the server return NFS4ERR_LAYOUTUNAVAILABLE, then the client
   proceeds to read from the file through the metadata server in the
   usual manner.  Otherwise, the client interprets the returned file
   layout in the manner specified by Section 13 of
   [draft-ietf-nfsv4-minorversion1-29] (NFSv4.1 as a Storage Protocol in
   pNFS: the File Layout Type).

2.2.  Meta data server responsibilities

   The metadata server has the usual responsibilities as dictated by
   Section 13 of [draft-ietf-nfsv4-minorversion1-29].  It maintains the
   list of available data servers for each file, and manages the layout
   requests from pNFS clients, responds to PROXY_OPEN requests from data
   servers, and ensures that PROXY_OPEN stateids are revoked when the
   corresponding layout is revoked.

2.3.  NFS client acting as a pNFS data server

   A client that wishes to act as a data server is required to notify
   the metadata server of that intention using the REGISTER_DS
   operation.  Depending on the circumstances, the client may opt to
   register as a data server for all cached files, for just a single
   filesystem, for a collection of filesystems, for a collection of
   specific files, or for just a single file.

   As stated in the introduction, the design assumes that the sharing of
   cached data between NFS clients will reduce the amount of NFS traffic
   to the permanent storage medium.  It therefore only makes sense to
   invoke this model in the case when the server knows that the client
   that is acting as a data server is caching the file data
   aggressively.  In order to verify that this is the case, we require
   that the metadata server can only issue layouts for data servers that
   hold a read delegation for the file in question.

   Conversely, a client that is registered to act as a data server, and
   that receives a READ request for a file for which it does not hold a
   delegation, MUST reject that request with the error
   NFS4ERR_PNFS_NO_LAYOUT.

   When the data server receives a READ request from a client with a

Myklebust                Expires January 7, 2010                [Page 5]

Internet-Draft      pNFS back end protocol extensions          July 2009

   stateid or a data server filehandle that it does not recognise, it
   attempts to validate that request using the PROXY_OPEN call.  This
   operation will convert the data server filehandle as provided by the
   layout into a real filehandle, that the data server can use to access
   the file on the metadata server.  In order to make it easy for the
   data server to identify the file, the real filehandle SHOULD match
   the filehandle that was returned to the client when it received the
   read delegation.

   The PROXY_OPEN call also checks the access rights that were granted
   by the layout and the READ stateid for validity.  If the pNFS client
   in question does not hold a layout for this file, the PROXY_OPEN
   request from the data server will return NFS4ERR_PNFS_NO_LAYOUT.  In
   this case, the data server should not attempt to service the READ
   request, but should pass the error on to the pNFS client.

   If file access was verified by PROXY_OPEN, the data server can then
   attempt to service the READ request from its cache.  Should it fail
   to find the data in its cache, the data server should attempt to
   retrieve it from the parent server.

   When layouts are returned to the metadata server, the data server is
   made responsible for fencing off any further READ requests.  To do
   so, the metadata server sends a CB_PROXY_REVOKE callback to the data
   server (or servers) that are referenced by that layout.  Upon
   receiving the CB_PROXY_REVOKE callback, the data server should match
   the filehandle and stateid arguments to the data filehandle that was
   previously used as an argument to the PROXY_OPEN request, and the
   stateid that was returned by that request.  Should the client attempt
   to reuse the same data filehandle and stateid in a future READ
   request, then the data server SHOULD revalidate the client's access
   using another PROXY_OPEN rpc call to the metadata server.

   An NFS client can at all times revoke its offer to act as a data
   server by using the UNREGISTER_DS operation.  This operation takes a
   single stateid, as returned by the original REGISTER_DS request.
   When the metadata server receives such a request, it must immediately
   revoke all layouts that reference that particular data server.  It
   does not need to send a CB_PROXY_REVOKE notification to the data
   server that it unregistering, however it MUST any other data servers
   that are referenced by the same layout.

Myklebust                Expires January 7, 2010                [Page 6]

Internet-Draft      pNFS back end protocol extensions          July 2009

3.  Security considerations

   As per Section 13.1 in [draft-ietf-nfsv4-minorversion1-29], it is
   expected that metadata servers will need to encode server routing
   information in the data server filehandles.  To enable this, the
   REGISTER_DS request includes a 64-bit cookie argument that the
   metadata server is required to store.  It is then required to encode
   that 64-bit cookie in the first 64-bits of the data server
   filehandle.

   All operations from the data server to the metadata server, including
   any operations required to refill the file cache in order to satisfy
   a READ request by the pNFS client should be authenticated using a
   principal of the form "nfsd/hostname@REALM".  It is, however expected
   that this requirement will be obsoleted, should the proposal for
   RPCSEC_GSSv3 [draft-williams-rpcsecgssv3] be approved.  In this case,
   the data server may instead choose to create a process credential
   that asserts the credentials of the pNFS client.

Myklebust                Expires January 7, 2010                [Page 7]

Internet-Draft      pNFS back end protocol extensions          July 2009

4.  State expiration and recovery considerations

   Should the pNFS client's session expire on the metadata server, then
   the latter is required to recall all layouts from the data servers
   using the CB_PROXY_REVOKE callback.  Upon re-establishing the
   session, the pNFS client then proceeds to follow the usual state
   recovery routine, including layout recovery.

   Should the pNFS client's session expire on the data server then it is
   required to recover that session before it can issue a new READ
   request.  In that case, the data server MUST assume that all existing
   layouts have been revoked.  Should the pNFS client attempt to assert
   a layout then it MUST be validated using a PROXY_OPEN call.

   Should the data server's session expire on the metadata server, then
   the metadata server MUST revoke all layouts that reference that data
   server.  It should also consider as invalid any REGISTER_DS requests
   that the data server had issued.  After recovering its session, the
   data server MAY reissue the REGISTER_DS requests.

   Finally, if the metadata server crashes, then the data server SHOULD
   assert all REGISTER_DS requests as part of the recovery process.
   Once that is done, it must also assume that all layouts have been
   revoked, and that any attempt to reuse them MUST be revalidated using
   a PROXY_OPEN request.  Otherwise, both it and the pNFS client perform
   the normal NFS client recovery process.

Myklebust                Expires January 7, 2010                [Page 8]

Internet-Draft      pNFS back end protocol extensions          July 2009

5.  Structured Data Types

5.1.  proxy_identifier4

     union proxy_identifier4 switch (uint32_t flavor) {
     case RPCSEC_GSS:
           principal_arg           pid_principal;
     case AUTH_SYS:
           struct authsys_parms    pid_authsys;
     default:
           void;
     };

   The proxy_identifier4 data type is used to identify the user on
   behalf of which the data server is issuing a PROXY_OPEN.

Myklebust                Expires January 7, 2010                [Page 9]

Internet-Draft      pNFS back end protocol extensions          July 2009

6.  New client operations

6.1.  REGISTER_DS - Offer to act as a data server

6.1.1.  ARGUMENTS

     const NFS4_MDS_IDENTIFIER_SIZE       = 8;

     enum register_ds_type4 {
           REGISTER_DS_ALL                 = 0,
           REGISTER_DS_FILESYSTEM          = 1,
           REGISTER_DS_ADD_FILESYSTEM      = 2
           REGISTER_DS_FILE                = 3
           REGISTER_DS_ADD_FILE            = 4
     };

     typedef opaque mds_identifier4[NFS4_MDS_IDENTIFIER_SIZE];

     union register_ds (register_ds_type4 ds_type) {
     case REGISTER_DS_ALL:
           mds_identifier4 rea_mds_identifier;
     case REGISTER_DS_FILESYSTEM:
           /* CURRENT_FH: file on filesystem being re-exported */
           mds_identifier4 rea_mds_identifier;
     case REGISTER_DS_ADD_FILESYSTEM:
           /* CURRENT_FH: file on filesystem being re-exported */
           stateid4        rea_dataserver_stateid;
     case REGISTER_DS_FILE:
           /* CURRENT_FH: file being re-exported */
           mds_identifier4 rea_mds_identifier;
     case REGISTER_DS_ADD_FILE:
           /* CURRENT_FH: file being re-exported */
           stateid4        rea_dataserver_stateid;
     };

     struct REGISTER_DS4args {
           register_ds     rea_dsinfo;
     };

6.1.2.  RESULTS

     union REGISTER_DS4res (nfsstat4 status) {
     case NFS4_OK:
           stateid4        res_dataserver_stateid;
     default:
           void;
     };

Myklebust                Expires January 7, 2010               [Page 10]

Internet-Draft      pNFS back end protocol extensions          July 2009

6.1.3.  DESCRIPTION

   The REGISTER_DS operation allows an NFS client to signal to the
   metadata server its ability to act as a data server towards other NFS
   clients.  Note that the MDS MUST not ever issue a layout for a file
   for which the client does not hold a read delegation.

   The client can register an intention to export all files for which it
   holds a read delegation, using the argument REGISTER_DS_ALL.

   It is also anticipated that some NFS setups may have the ability to
   set a caching and/or re-exporting policy.  For such setups, it is
   possible to set more fine-grained data server policies:
      REGISTER_DS_FILESYSTEM allows the client to specify that it wants
      to be a data server for a specific filesystem only.
      REGISTER_DS_ADD_FILESYSTEM allows the client to specify that it
      wants to add a filesystem to the data server represented by the
      stateid 'rea_dataserver_stateid'.
      REGISTER_DS_FILE allows the client to specify that it wants to
      serve a particular file only.
      REGISTER_DS_ADD_FILE allows the client to specify that it wants to
      add a file to the data server represented by the stateid
      'rea_dataserver_stateid'.

   The client should also supply a unique 64-bit identifier in the
   argument rea_mds_identifier.  This identifier should be put as the
   first 8 bytes of any data server filehandle, and may be used by the
   data server to identify the MDS to which the filehandle belongs.

   On success, the server returns the stateid 'res_dataserver_stateid'
   which acts to identify the data server in future REGISTER_DS calls,
   and in UNREGISTER_DS calls.

   The client may in fact hold several data server stateids, and use
   them to manage the overall policy.

6.2.  UNREGISTER_DS - Revoke offer to act as a data server

6.2.1.  ARGUMENTS

     struct UNREGISTER_DS4args {
           stateid4        una_dataserver_stateid;
     };

Myklebust                Expires January 7, 2010               [Page 11]

Internet-Draft      pNFS back end protocol extensions          July 2009

6.2.2.  RESULTS

     struct UNREGISTER_DS4res {
           nfsstat4        status;
     };

6.2.3.  DESCRIPTION

   When the MDS receives an UNREGISTER_DS operation then it must
   immediately invalidate all state associated with the data server
   stateid 'una_dataserver_stateid'.

   It MUST therefore revoke all layouts that refer to the data server
   that is represented by una_dataserver_stateid.

   After revoking the layouts, the MDS MUST no longer issue layouts for
   these files and filesystems using the data server represented by
   una_dataserver_stateid.

6.3.  PROXY_OPEN - Check proxy access rights to a file

   This takes a data server filehandle, the read stateid, and a proxy
   user identifier, and returns the true filehandle on success.

6.3.1.  ARGUMENTS

     struct PROXY_OPEN4args {
           /* CURRENT_FH: "data server filehandle" */
           proxy_identifier4       popa_user_id;
           stateid4                popa_read_stateid;
     };

6.3.2.  RESULTS

     union PROXY_OPEN4res switch (nfsstat4 status) {
           case NFS4_OK:
                   /* CURRENTFH: true filehandle */
                   stateid4        popr_proxy_stateid;
           default:
                   void;
     };

6.3.3.  DESCRIPTION

   The PROXY_OPEN function authenticates the READ request by the pNFS
   client.  If the data filehandle is valid, and the user identified by
   the popa_user_id is authorised to access the file, then the metadata
   server returns the true filehandle (as returned by LOOKUP and/or

Myklebust                Expires January 7, 2010               [Page 12]

Internet-Draft      pNFS back end protocol extensions          July 2009

   OPEN) of the file.

   If the pNFS client does not currently hold a layout for this file,
   then the PROXY_OPEN request should fail with the error
   NFS4ERR_PNFS_NO_LAYOUT.

   If the data server filehandle argument cannot be translated into a
   valid metadata server filehandle, then the errors NFS4ERR_STALE,
   NFS4ERR_BADHANDLE, or NFS4ERR_FHEXPIRED should be returned, as
   appropriate.

   If the stateid argument does not correspond to a valid open stateid,
   delegation stateid, or lock stateid, for the file that is being
   attempted READ, then the metadata server should return the
   appropriate error.

   In case of success, the metadata server returns a stateid
   "popr_proxy_stateid" that is used by the CB_PROXY_REVOKE callback to
   identify which layout is being revoked.

Myklebust                Expires January 7, 2010               [Page 13]

Internet-Draft      pNFS back end protocol extensions          July 2009

7.  New callback operations

7.1.  CB_PROXY_REVOKE - revoke proxy access rights to a file

   This takes a data server filehandle, and a proxy open stateid, and
   revokes them.

7.1.1.  ARGUMENTS

     struct CB_PROXY_REVOKE4args {
           nfs_fh4         pra_object;
           stateid4        pra_proxy_stateid;
     };

7.1.2.  RESULTS

     struct CB_PROXY_REVOKE4res{
           nfsstat4        prr_status;
     };

7.1.3.  DESCRIPTION

   pra_object is the data server filehandle for the file, whereas
   pra_proxy_stateid is the stateid that was returned by the PROXY_OPEN
   operation.

   Upon receiving this callback, the data server MUST invalidate all
   state associated with the stateid pra_proxy_stateid, and return
   NFS4_OK.

   If the filehandle was not found, the client MUST return
   NFS4ERR_BADHANDLE.  If the stateid was not found, it MUST return
   NFS4ERR_BAD_STATEID.

Myklebust                Expires January 7, 2010               [Page 14]

Internet-Draft      pNFS back end protocol extensions          July 2009

8.  References

   [RFC2119]  Bradner, S., "Key words for use in RFCs to Indicate
              Requirement Levels", RFC 2119.

   [draft-ietf-nfsv4-minorversion1-29]
              Shepler, S., Eisler, M., and D. Noveck, "NFS Version 4
              Minor Version 1", draft-ietf-nfsv4-minorversion1 29.

   [draft-williams-rpcsecgssv3]
              Williams, N., "Remote Procedure Call (RPC) Security
              Version 3", draft-williams-rpcsecgssv3 00.

Myklebust                Expires January 7, 2010               [Page 15]

Internet-Draft      pNFS back end protocol extensions          July 2009

Author's Address

   Trond Myklebust
   NetApp
   3215 Bellflower Ct
   Ann Arbor, MI  48103
   USA

   Phone: +1-734-662-6608
   Email: [email-masked]

Myklebust                Expires January 7, 2010               [Page 16]

Internet-Draft      pNFS back end protocol extensions          July 2009

Full Copyright Statement

   Copyright (C) The IETF Trust (2009).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   [email-masked].

Myklebust                Expires January 7, 2010               [Page 17]

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Jul 2009 21:18:04 -0500

You should also provide the XDR in the form of a dot-x file, yes?

A broader concern then becomes how do we manage the Control Protocol 
space (including versions)
such that different implementations do not walk over each other.

---

Section 2.3.

>    that are referenced by the same layout.

it MUST do what?

I think your intent here is to say that if a caching DS is used in a 
layout that spans caching DSes, then
those caching DSes must be informed of the layout recall.

---

How does the caching DS know that the MDS has rebooted?

As I see it, that can only happen on a PROXY_OPEN or if the READ fails 
to find anything in the cache.

So the caching DS could happily hand out "stale" information until the 
state expires. Are you thinking
that these PROXY_OPENs are going to occur frequently or that the cache 
is small?

OR are you counting on regular 4.1 procedures to detect the MDS rebooting?

---

Section 4:

>    recovery routine, including layout recovery.

Why force the client to do state recovery when the client could simply 
extend the session?

OR are you counting on regular 4.1 procedures to keep the session fresh?

---

So a light may have clicked about using 4.1 procedures, but it doesn't 
hurt to call these out. I.e.,
terseness leads to interpretation.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 06 Jul 2009 23:29:11 -0400

> You should also provide the XDR in the form of a dot-x file, yes?

A dot-what? Does anybody still use those? What on earth for?

> such that different implementations do not walk over each other.

I'm not sure that I understand your concern. Why would we want to
introduce control protocols that might conflict with each other?

> those caching DSes must be informed of the layout recall.

Yes. The above should read as

"It does not need to send a CB_PROXY_REVOKE notification to the data
server that it is unregistering, however it MUST notify any other data
servers that are referenced by the same layout."

> How does the caching DS know that the MDS has rebooted?

The usual way: the data server is also an NFS client which is holding
state, so it is required to ping the metadata server with a SEQUENCE op
every lease period or so.

> terseness leads to interpretation.

I don't intend to change the behaviour of pNFS when the session is lost
on the metadata server. That paragraph was added simply for symmetry
with the other recovery cases, and is supposed to mirror the behaviour
described in the NFSv4.1 draft spec.

Cheers
  Trond

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Jul 2009 22:58:44 -0500

> introduce control protocols that might conflict with each other?

And that goes back to the dot-x.

Who dictates what goes into this uber-Control Protocol and how do we 
detect what is
and is not supported? The base spec is pretty loose in describing all of 
this.

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 07 Jul 2009 07:33:41 -0400

> And that goes back to the dot-x.

It does?

> and is not supported? The base spec is pretty loose in describing all of 

Now you have me really confused. The control protocol _I_ am proposing
is a limited scheme that wants to be integrated into the NFSv4 protocol
in order to allow a very particular mode of file sharing between NFS
clients. See the introduction.
It is not designed to be an "uber-control protocol" to obsolete all
existing private control protocols.

If someone else has a proposal for a more complete control protocol and
can justify that it needs to be integrated into the main NFSv4 protocol
(the rough consensus for NFSv4.1 was that control protocols were a
separate issue), then there is plenty of opportunity for doing so. One
can make a case for it now, by writing an RFC and attempting to achieve
rough consensus, or if said proposal doesn't make the deadline for
NFSv4.2, then we have the minor versioning scheme to help integrate it
later: just look up RFC3530 and the draft NFSv4.1 spec for details on
how to prepare an NFSv4.3.

Trond

---

From: Tom Haynes <[email-masked]>
Date: Fri, 10 Jul 2009 11:10:59 -0500

> how to prepare an NFSv4.3.

My point was simply that we should think about laying such groundwork 
before we introduced
the first control protocol variant.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 10 Jul 2009 19:34:57 -0400

> the first control protocol variant.

As I remember it, the consensus when we discussed NFSv4.1 was that
control protocols are going to be intimately tied to the specifics of
the server topology and design. Furthermore, since they are not part of
the client-server interaction, they lie outside the scope of the NFS
spec. Now if that consensus has changed since we published the 4.1
draft, then I'd be happy to revisit the topic.

My justification for introducing a control protocol here is that in
order to allow the NFS server to issue layouts for a client that is
acting as a data server, then the two do need a common protocol for
negotiating the data server role.
While we could indeed set this up as a private side-band protocol, I
think there is merit in merging it into the mainstream NFS protocol in
order to improve the scalability qualities of the latter.

Please note that most of the functions that are introduced are very
generic:

      * REGISTER_DS/UNREGISTER_DS are going to be needed for pretty much
        any control protocol.
      * The PROXY_OPEN addresses a file access and authentication issue
        that affects all pNFS files based data servers, although it is
        quite possible that the list of arguments is insufficient to
        address all future use cases. I'm quite open to suggestions...
      * CB_PROXY_REVOKE is just basic fencing for a pNFS files based
        data server. It revokes the file access authorisation that was
        previously granted. Again, this is something your control
        protocol will always require.

      Trond

---

From: Tom Haynes <[email-masked]>
Date: Mon, 13 Jul 2009 12:42:58 -0500

Trond Myklebust wrote:
> negotiating the data server role

I think I'm causing a rat-hole storm here.

I don't have any objection with your control protocol proposal.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 13 Jul 2009 17:44:43 -0400

>    It is not obvious that use of the pNFS scale out mode is sufficient

Strike "scale out mode".

>    the bottleneck from the metadata server over onto the data servers.

s/over onto/to/.

(Also: that seems a little too quickly dismissed.  Both data servers and
client/data-servers provide more aggregate bandwidth than a single MDS,
and it's not obvious in which case one will provide more than the
other.)

>    require the user to dedicate extra resources for the boot process.

s/resources/hardware resources/ ?

>    corresponding layout is revoked.

The first two in that list are among the "usual responsibilities"; the
last two are new.  Maybe make that: "... from pNFS clients.  In
addition, it responds to PROXY_OPEN requests..."

>    specific files, or for just a single file.

Remove both "just"s.

>    As stated in the introduction, the design assumes that the sharing of

s/the sharing/sharing/.

>    that the metadata server can only issue layouts for data servers that

s/can //
s/for/to/.

>    hold a read delegation for the file in question.

I know you're interested in read-only caching, but is there any reason a
write delegation wouldn't be sufficient for that?

>    delegation, MUST reject that request with the error

If the client/data-server wants to return its delegation, is there some
way it can do so without later having to return NFS4ERR_PNFS_NO_LAYOUT
to clients that believe they still do hold valid layouts?  (Perhaps
that's not a problem.  Just curious.)

Reading further down: I suppose the client should send an UNREGISTER_DS
before returning a delegation?

>    the filehandle that was returned to the client when it received the

(So the lack of a MUST allows the metadata server to return a different
filehandle, as long as the client can still check that it points to the
same file (by checking the fileid, etc.)?  OK.)

>    to reuse the same data filehandle and stateid in a future READ

OK, so this is a data server filehandle, not a "real" filehandle?

>    server that it unregistering, however it MUST any other data servers

s/it/it is/.

s/MUST/MUST send a CB_PROXY_REVOKE to/ ?

>    that 64-bit cookie in the first 64-bits of the data server

s/64-bits/64 bits/.

Could you add some discussion of how this cookie is meant to be used?

I assume the idea is to give the client/data-server a way to determine
which filehandles belong to which metadata server (in case it has a
data-server relationship with multiple metadata servers), and which are
its own filehandles (in case it also happens to be an ordinary nfs
server or metadata server itself).  And the client has to choose the
cookie itself since it is the one that has to arrange for the cookie to
disambiguate those cases.  OK.

(Coming back: OK, I see that's explained below.  A little more
explanation on the first discussion might have helped, though.)

>    a READ request by the pNFS client should be authenticated using a

s/client/client,/

>    principal of the form "nfsd/hostname@REALM".

What does the client do about cached data that was acquired using the
credential of some user on that client?  Is it required to prefer
nfsd/hostname@REALM for any reads for which that is possible?

>It is, however expected
...
>            principal_arg           pid_principal;

What does a "principal_arg" look like on the wire?

>    clients.  Note that the MDS MUST not ever issue a layout for a file

s/MUST not/MUST NOT/.

>    argument rea_mds_identifier.  This identifier should be put as the

s/should/must/?

>    data server to identify the MDS to which the filehandle belongs.

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 13 Jul 2009 18:26:40 -0400

> write delegation wouldn't be sufficient for that?

Under normal NFSv4 rules, the write delegation is immediately revoked
when another client opens a file for READ.

That said, it is interesting to theorise that an NFS client that is
writing to a file should still be able to cache those writes
aggressively, provided that anyone who wants to read the file data do so
by using the writing client as the pNFS data server.
In other words, it might be possible to allow a relaxation of the rules
that OPEN always revokes a write delegation, _if_ the NFS client
volunteers to act as a data server.

The main reason why I haven't pursued this idea in the current draft is
that I'm missing a workload that could make use of such a property.

> that's not a problem.  Just curious.)

If the data server returns the delegation, then the metadata server is
supposed to automatically revoke all layouts, on the assumption that
we've lost the guarantee that this file is being cached aggressively.

> before returning a delegation?

No. There should be no reason for that as long as we adhere to the
convention that the server may only issue layouts for files that are
delegated.

> same file (by checking the fileid, etc.)?  OK.)

Yes.

> OK, so this is a data server filehandle, not a "real" filehandle?

Sorry, yes. The above sentence should read:

"Upon receiving the CB_PROXY_REVOKE callback, the data server should
match the filehandle and stateid arguments to the data server filehandle
that was previously used as an argument to the PROXY_OPEN request..."

> explanation on the first discussion might have helped, though.)

Sure. I can move the discussion around a bit.

> nfsd/hostname@REALM for any reads for which that is possible?

Authentication is performed at PROXY_OPEN time. If ACLs or file
ownership or mode bits change, then I suppose the metadata server should
revoke the PROXY_OPEN. I can add a section about that case...

> What does a "principal_arg" look like on the wire?

I've always thought of it as just being the string name. I believed that
it was defined in the NFSv4.1 draft spec, however the latter just says
that it is "The value of the RPCSEC_GSS principal for the current
request".

The other changes you propose look trivially correct, so I'm updating
the draft.

Cheers
  Trond

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 13 Jul 2009 19:17:37 -0400

> when another client opens a file for READ.

Oh, right, so the client that might benefit from this cache is never
going to get to use it since its open immediate revokes the write
delegation held by its data server.... I missed that.

> that I'm missing a workload that could make use of such a property.

OK, that seems reasonable.

> revoke the PROXY_OPEN. I can add a section about that case...

I was curious about the case where the client reads a file as
malloy@REALM, then returns the cached data to Alice, who is on another
client.  Then Alice is trusting Malloy, to some extent, since Malloy may
(assuming access to the network, etc.) have had the ability to fake the
file data.

The same could have happened if Alice and Malloy shared the same client,
but this extends the attack a little further.  (Alice may now have to
trust users on other clients where previously she only had to trust
users on her client.)

Also related: if Alice uses krb5i or krb5p, but the original
metadata-to-data-server reads use krb5, then (depending on the
situation), the purpose of the integrity or privacy may be defeated.
(Which could also happen on a single client depending on implementation
and administrator policies.)

I doubt there's any fatal problem hidden here.  I just wonder if the
final draft of this may need some discussion which wasn't needed in the
pNFS spec since any data-server<->metadata-server security was out of
scope.

--b.