Subject: [nfsv4] Fwd: draft-ietf-nfsv4-pnfs-obj-04

From: Benny Halevy <[email-masked]>
Date: Wed, 05 Sep 2007 22:15:49 +0300

I submitted draft-ietf-nfsv4-pnfs-obj-04.
Please see summary of changes below.

Benny

-------- Original Message --------
Subject: 	draft-ietf-nfsv4-pnfs-obj-04
Date: 	Wed, 05 Sep 2007 22:13:20 +0300
From: 	Benny Halevy <[email-masked]>
To: 	[email-masked]
CC: 	Benny Halevy <[email-masked]>
References: 	<[email-masked]>

Attached is an updated I-D, draft-ietf-nfsv4-pnfs-obj-04.txt
and its corresponding source file, draft-ietf-nfsv4-pnfs-obj-04.xml.

It updates the nfsv4 working group I-D, draft-ietf-nfsv4-pnfs-obj-03.txt

The new version has several editorial changes and clarifications
as well these technical changes:

- Added optional System ID to pnfs_osd_deviceaddr4.
  Clarified its use in the system and how it is related
  to the OSD name.

- Added cap_key_sec to pnfs_osd_object_cred4 specifying the
  method used to secure the capability key.  Specified
  the different methods.

- Broke the opaque pnfs_osd_object_cred4.credential into two
  opaque fields: capability_key and capability since the pnfs client
  must be able to parse the internal structure of the credential.

- Reorganized and clarified pnfs_osd_errno4 error codes.
  Added PNFS_OSD_ERR_RESOURCE.

- Specified object-based layout specific mask flags for CB_RECALL_ALL.

- Added Client Fencing section.

- Added paragraph to Security Considerations describing the method
  used to enforce the server's access control policy and the possibility
  of rejecting LAYOUTGET with NFS4ERR_ACCESS.

- Disallowed the used of NOSEC OSD security method for anything but
  the first OSD GET ATTRIBUTE the might be needed to read the OSD
  System ID.

- Separated the protocol privacy requirements as a sub-section in
  Security Considerations.

- Corrected Revoking Capabilities section.

--
Benny Halevy
Software Architect
Tel/Fax: +972-3-647-8340
Mobile: +972-54-802-8340
US:      +1-412-203-3187
[email-masked]

Panasas, Inc.
Leader in Parallel Storage
www.panasas.com

NFSv4                                                          B. Halevy
Internet-Draft                                                  B. Welch
Intended status: Standards Track                              J. Zelenka
Expires: March 8, 2008                                           Panasas
                                                       September 5, 2007

                      Object-based pNFS Operations
                      draft-ietf-nfsv4-pnfs-obj-04

Status of this Memo

   By submitting this Internet-Draft, each author represents that any
   applicable patent or other IPR claims of which he or she is aware
   have been or will be disclosed, and any of which he or she becomes
   aware will be disclosed, in accordance with Section 6 of BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on March 8, 2008.

Copyright Notice

   Copyright (C) The IETF Trust (2007).

Abstract

   This Internet-Draft provides a description of the object-based pNFS
   extension for NFSv4.  This is a companion to the main pnfs
   specification in the NFSv4 Minor Version 1 Internet Draft, which is
   currently draft-ietf-nfsv4-minorversion1-13.txt.

Halevy, et al.            Expires March 8, 2008                 [Page 1]

Internet-Draft                pnfs objects                September 2007

Requirements Language

   The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
   "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
   document are to be interpreted as described in RFC 2119 [1].

Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  4
   2.  Object Storage Device Addressing and Discovery . . . . . . . .  4
     2.1.  pnfs_osd_addr_type4  . . . . . . . . . . . . . . . . . . .  5
     2.2.  pnfs_osd_deviceaddr4 . . . . . . . . . . . . . . . . . . .  6
   3.  Object-Based Layout  . . . . . . . . . . . . . . . . . . . . .  6
     3.1.  pnfs_osd_layout4 . . . . . . . . . . . . . . . . . . . . .  7
       3.1.1.  pnfs_osd_objid4  . . . . . . . . . . . . . . . . . . .  7
       3.1.2.  pnfs_osd_version4  . . . . . . . . . . . . . . . . . .  8
       3.1.3.  pnfs_osd_object_cred4  . . . . . . . . . . . . . . . .  8
       3.1.4.  pnfs_osd_raid_algorithm4 . . . . . . . . . . . . . . . 10
       3.1.5.  pnfs_osd_data_map4 . . . . . . . . . . . . . . . . . . 10
     3.2.  Data Mapping Schemes . . . . . . . . . . . . . . . . . . . 11
       3.2.1.  Simple Striping  . . . . . . . . . . . . . . . . . . . 11
       3.2.2.  Nested Striping  . . . . . . . . . . . . . . . . . . . 12
       3.2.3.  Mirroring  . . . . . . . . . . . . . . . . . . . . . . 13
     3.3.  RAID Algorithms  . . . . . . . . . . . . . . . . . . . . . 14
       3.3.1.  PNFS_OSD_RAID_0  . . . . . . . . . . . . . . . . . . . 14
       3.3.2.  PNFS_OSD_RAID_4  . . . . . . . . . . . . . . . . . . . 14
       3.3.3.  PNFS_OSD_RAID_5  . . . . . . . . . . . . . . . . . . . 15
       3.3.4.  PNFS_OSD_RAID_PQ . . . . . . . . . . . . . . . . . . . 15
       3.3.5.  RAID Usage and implementation notes  . . . . . . . . . 16
   4.  Object-Based Layout Update . . . . . . . . . . . . . . . . . . 16
     4.1.  pnfs_osd_layoutupdate4 . . . . . . . . . . . . . . . . . . 16
       4.1.1.  pnfs_osd_deltaspaceused4 . . . . . . . . . . . . . . . 17
       4.1.2.  pnfs_osd_errno4  . . . . . . . . . . . . . . . . . . . 17
       4.1.3.  pnfs_osd_ioerr4  . . . . . . . . . . . . . . . . . . . 18
   5.  Object-Based Creation Layout Hint  . . . . . . . . . . . . . . 19
     5.1.  pnfs_osd_layouthint4 . . . . . . . . . . . . . . . . . . . 19
   6.  Layout Segments  . . . . . . . . . . . . . . . . . . . . . . . 20
     6.1.  CB_LAYOUTRECALL and LAYOUTRETURN . . . . . . . . . . . . . 20
     6.2.  LAYOUTCOMMIT . . . . . . . . . . . . . . . . . . . . . . . 21
   7.  Recalling Layouts  . . . . . . . . . . . . . . . . . . . . . . 21
     7.1.  CB_RECALL_ANY  . . . . . . . . . . . . . . . . . . . . . . 22
   8.  Client Fencing . . . . . . . . . . . . . . . . . . . . . . . . 22
   9.  Security Considerations  . . . . . . . . . . . . . . . . . . . 23
     9.1.  OSD Security Data Types  . . . . . . . . . . . . . . . . . 24
     9.2.  The OSD Security Protocol  . . . . . . . . . . . . . . . . 24
     9.3.  Protocol Privacy Requirements  . . . . . . . . . . . . . . 25
     9.4.  Revoking Capabilities  . . . . . . . . . . . . . . . . . . 26

Halevy, et al.            Expires March 8, 2008                 [Page 2]

Internet-Draft                pnfs objects                September 2007

   10. IANA Considerations  . . . . . . . . . . . . . . . . . . . . . 27
   11. References . . . . . . . . . . . . . . . . . . . . . . . . . . 27
     11.1. Normative References . . . . . . . . . . . . . . . . . . . 27
     11.2. Informative References . . . . . . . . . . . . . . . . . . 27
   Appendix A.  Acknowledgments . . . . . . . . . . . . . . . . . . . 27
   Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . . 28
   Intellectual Property and Copyright Statements . . . . . . . . . . 29

Halevy, et al.            Expires March 8, 2008                 [Page 3]

Internet-Draft                pnfs objects                September 2007

1.  Introduction

   In pNFS, the file server returns typed layout structures that
   describe where file data is located.  There are different layouts for
   different storage systems and methods of arranging data on storage
   devices.  This document describes the layouts used with object-based
   storage devices (OSD) that are accessed according to the iSCSI/OSD
   storage protocol standard (SNIA T10/1355-D [2]).

   An "object" is a container for data and attributes, and files are
   stored in one or more objects.  The OSD protocol specifies several
   operations on objects, including READ, WRITE, FLUSH, GET ATTRIBUTES,
   SET ATTRIBUTES, CREATE and DELETE.  However, using the object-based
   layout the client only uses the READ, WRITE, GET ATTRIBUTES and FLUSH
   commands.  The other commands are only used by the pNFS server.

   An object-based layout for pNFS includes object identifiers,
   capabilities that allow clients to READ or WRITE those objects, and
   various parameters that control how file data is striped across their
   component objects.  The OSD protocol has a capability-based security
   scheme that allows the pNFS server to control what operations and
   what objects can be used by clients.  This scheme is described in
   more detail in the Security Considerations section (Section 9).

2.  Object Storage Device Addressing and Discovery

   Data operations to an OSD require the client to know the "address" of
   each OSD's root object.  The root object is synonymous with SCSI
   logical unit.  The client specifies SCSI logical units to its SCSI
   stack using a representation local to the client.  Because these
   representations are local, GETDEVICEINFO must return information that
   can be used by the client to select the correct local representation.

   In the block world, a set offset (logical block number or track/
   sector) contains a disk label.  This label identifies the disk
   uniquely.  In contrast, an OSD has a standard set of attributes on
   its root object.  For device identification purposes the OSD System
   ID (root information attribute number 3) and/or OSD Name (root
   information attribute number 9) are used as the label.  These appear
   in the pnfs_osd_deviceaddr4 type below under the "systemid" and
   "osdname" fields.

   In some situations, SCSI target discovery may need to be driven based
   on information contained in the GETDEVICEINFO response.  One example
   of this is iSCSI targets that are not known to the client until a
   layout has been requested.  Eventually iSCSI will adopt ANSI T10
   SAM-3, at which time the World Wide Name (WWN aka, EUI-64/EUI-128)

Halevy, et al.            Expires March 8, 2008                 [Page 4]

Internet-Draft                pnfs objects                September 2007

   naming conventions can be specified.  In addition, Fibre Channel (FC)
   SCSI targets have a unique WWN.  Although these FC targets have
   already been discovered, some implementations may want to specify the
   WWN in addition to the label.  This information appears as the
   "target" and "lun" fields in the pnfs_osd_deviceaddr4 type described
   below.

   The systemid is used by the client, along with the object credential
   to sign each request with the request integrity check value.  This
   method protects the client from unintentionally accessing a device if
   the device address mapping was changed (or revoked).  The server
   computes the capability_key using its own view of the systemid
   associated with the respective deviceid present in the credential.
   If the client's view of the deviceid mapping is stale, the client
   will use the wrong systemid (which must be system-wide unique) and
   the I/O request to the OSD will fail to pass the integrity check
   verification.

   To recover from this condition the client should report the error via
   LAYOUTCOMMIT, return the layout using LAYOUTRETURN, and invalidate
   all the device address mappings associated with this layout.  The
   client can then ask for a new layout if it wishes using LAYOUTGET and
   resolve the referenced deviceids using GETDEVICEINFO or
   GETDEVICELIST.

   The server MUST provide either the systemid, the OSD name, or both.
   When the OSD name is present the client SHOULD get the root
   information attributes whenever it establishes communication with the
   OSD and verify that the OSD name it got from the OSD matches the one
   sent by the metadata server.  If the systemid was not given by the
   server it MUST be taken from the OSD-provided attribute; note that in
   this case the OSD GET ATTRIBUTES operation must be performed with the
   NOSEC security method.

2.1.  pnfs_osd_addr_type4

   The following enum specifies the manner in which a scsi target can be
   specified.  The target can be specified as a network address, as an
   Internet Qualified Name (IQN), or by the World-Wide Name (WWN) of the
   target.

   enum pnfs_obj_addr_type4 {
       OBJ_TARGET_NETADDR  = 1,
       OBJ_TARGET_IQN      = 2,
       OBJ_TARGET_WWN      = 3
   };

Halevy, et al.            Expires March 8, 2008                 [Page 5]

Internet-Draft                pnfs objects                September 2007

2.2.  pnfs_osd_deviceaddr4

   The specification for an object device address is as follows:

   struct pnfs_osd_deviceaddr4 {
       union target switch (pnfs_osd_addr_type4 type) {
           case OBJ_TARGET_NETADDR:
               pnfs_netaddr4   netaddr;

           case OBJ_TARGET_IQN:
               string          iqn<>;

           case OBJ_TARGET_WWN:
               string          wwn<>;

           default:
               void;
       };
       uint64_t            lun;
       opaque              systemid<>;
       opaque              osdname<>;
   };

3.  Object-Based Layout

   The layout4 type is defined in the NFSv4.1 draft [6] as follows:

   enum layouttype4 {
       LAYOUT4_NFSV4_1_FILES   = 1,
       LAYOUT4_OSD2_OBJECTS    = 2,
       LAYOUT4_BLOCK_VOLUME    = 3
   };

   struct layout_content4 {
       layouttype4             loc_type;
       opaque                  loc_body<>;
   };

   struct layout4 {
       offset4                 lo_offset;
       length4                 lo_length;
       layoutiomode4           lo_iomode;
       layout_content4         lo_content;
   };

   This document defines structure associated with the layouttype4

Halevy, et al.            Expires March 8, 2008                 [Page 6]

Internet-Draft                pnfs objects                September 2007

   value, LAYOUT4_OSD2_OBJECTS.  The NFSv4.1 draft [6] specifies the
   loc_body structure as an XDR type "opaque".  The opaque layout is
   uninterpreted by the generic pNFS client layers, but obviously must
   be interpreted by the object-storage layout driver.  This document
   defines the structure of this opaque value, pnfs_osd_layout4.

3.1.  pnfs_osd_layout4

   struct pnfs_osd_layout4 {
       pnfs_osd_data_map4      map;
       pnfs_osd_object_cred4   components<>;
   };

   The pnfs_osd_layout4 structure specifies a layout over a set of
   component objects.  The components field is an array of object
   identifiers and security credentials that grant access to each
   object.  The organization of the data is defined by the
   pnfs_osd_data_map4 type that specifies how the file's data is mapped
   onto the component objects (i.e., the striping pattern).  The data
   placement algorithm that maps file data onto component objects assume
   that each component object occurs exactly once in the array of
   components.  Therefore, component objects MUST appear in the
   component array only once.

   Note that the layout depends on the file size, which the client
   learns from the generic return parameters of LAYOUTGET, by doing
   GETATTR commands to the metadata server.  The client uses the file
   size to decide if it should fill holes with zeros, or return a short
   read.  Striping patterns can cause cases where component objects are
   shorter than other components because a hole happens to correspond to
   the last part of the component object.

3.1.1.  pnfs_osd_objid4

   An object is identified by a number, somewhat like an inode number.
   The object storage model has a two level scheme, where the objects
   within an object storage device are grouped into partitions.

   struct pnfs_osd_objid4 {
       deviceid4       device_id;
       uint64_t        partition_id;
       uint64_t        object_id;
   };

   The pnfs_osd_objid4 type is used to identify an object within a
   partition on a specified object storage device. "device_id" selects
   the object storage device from the set of available storage devices.
   The device is identified with the deviceid4 type, which is an index

Halevy, et al.            Expires March 8, 2008                 [Page 7]

Internet-Draft                pnfs objects                September 2007

   into addressing information about that device returned by the
   GETDEVICELIST and GETDEVICEINFO pnfs operations.  Within an OSD, a
   partition is identified with a 64-bit number, "partition_id".  Within
   a partition, an object is identified with a 64-bit number,
   "object_id".  Creation and management of partitions is outside the
   scope of this standard, and is a facility provided by the object
   storage file system.

3.1.2.  pnfs_osd_version4

   enum pnfs_osd_version4 {
       PNFS_OSD_MISSING    = 0,
       PNFS_OSD_VERSION_1  = 1,
       PNFS_OSD_VERSION_2  = 2
   };

   The osd_version is used to indicate the OSD protocol version or
   whether an object is missing (i.e., unavailable).  Some layout
   schemes encode redundant information and can compensate for missing
   components, but the data placement algorithm needs to know what parts
   are missing.

   At this time the OSD standard is at version 1.0, and we anticipate a
   version 2.0 of the standard ((SNIA T10/1729-D [7])).  The second
   generation OSD protocol has additional proposed features to support
   more robust error recovery, snapshots, and byte-range capabilities.
   Therefore, the OSD version is explicitly called out in the
   information returned in the layout.  (This information can also be
   deduced by looking inside the capability type at the format field,
   which is the first byte.  The format value is 0x1 for an OSD v1
   capability.  However, it seems most robust to call out the version
   explicitly.)

3.1.3.  pnfs_osd_object_cred4

   enum pnfs_osd_cap_key_sec4 {
       PNFS_OSD_CAP_KEY_SEC_NONE = 0,
       PNFS_OSD_CAP_KEY_SEC_SSV  = 1,
   };

   struct pnfs_osd_object_cred4 {
       pnfs_osd_objid4         object_id;
       pnfs_osd_version4       osd_version;
       pnfs_osd_cap_key_sec4   cap_key_sec;
       opaque                  capability_key<>;
       opaque                  capability<>;
   };

Halevy, et al.            Expires March 8, 2008                 [Page 8]

Internet-Draft                pnfs objects                September 2007

   The pnfs_osd_object_cred4 structure is used to identify each
   component comprising the file.  The object_id identifies the
   component object, the osd_version represents the osd protocol
   version, or whether that component is unavailable, and the capability
   and capability key, along with the systemid from the
   pnfs_osd_deviceaddr, provide the OSD security credentials needed to
   access that object.  The cap_key_sec value denotes the method used to
   secure the capability_key (see Section 9.1 for more details).

   To comply with the OSD security requirements the capability key
   SHOULD be transferred securely to prevent eavesdropping (see
   Section 9).  Therefore, a client SHOULD either issue the LAYOUTGET
   operation via RPCSEC_GSS with the privacy service or to previously
   establish an SSV for the sessions via the NFSv4.1 SET_SSV operation.
   The pnfs_osd_cap_key_sec4 type is used to identify the method used by
   the server to secure the capability key.

   o  PNFS_OSD_CAP_KEY_SEC_NONE denotes that the capability_key is not
      encrypted in which case the client SHOULD issue the LAYOUTGET
      operation with RPCSEC_GSS with the privacy service or the NFSv4.1
      transport should be secured by using methods that are external to
      NFSv4.1 like the use of IPSEC [8] for transporting the NFSV4.1
      protocol.

   o  PNFS_OSD_CAP_KEY_SEC_SSV denotes that the capability_key contents
      are encrypted using the SSV GSS context and the capability key as
      inputs to the GSS_Wrap() function (see [3]) with the conf_req_flag
      set to TRUE.  The client MUST use the secret SSV key as part of
      the client's GSS context to decrypt the capability key using the
      value of the capability_key field as the input_message to the
      GSS_unwrap() function.  Note that to prevent eavesdropping of the
      SSV key the client SHOULD issue SET_SSV via RPCSEC_GSS with the
      privacy service.

   The actual method chosen depends on whether the client established a
   SSV key with the server and whether it issued the LAYOUTGET operation
   with the RPCSEC_GSS privacy method.  Naturally, if the client did not
   establish a SSV key via SET_SSV the server MUST use the
   PNFS_OSD_CAP_KEY_SEC_NONE method.  Otherwise, if the LAYOUTGET
   operation was not issued with the RPCSEC_GSS privacy method the
   server SHOULD secure the capability_key with the
   PNFS_OSD_CAP_KEY_SEC_SSV method.  The server MAY use the
   PNFS_OSD_CAP_KEY_SEC_SSV method also when the LAYOUTGET operation was
   issued with the RPCSEC_GSS privacy method.

Halevy, et al.            Expires March 8, 2008                 [Page 9]

Internet-Draft                pnfs objects                September 2007

3.1.4.  pnfs_osd_raid_algorithm4

   enum pnfs_osd_raid_algorithm4 {
       PNFS_OSD_RAID_0     = 1,
       PNFS_OSD_RAID_4     = 2,
       PNFS_OSD_RAID_5     = 3,
       PNFS_OSD_RAID_PQ    = 4     /* Reed-Solomon P+Q */
   };

   pnfs_osd_raid_algorithm4 represents the data redundancy algorithm
   used to protect the file's contents.  See Section 3.3 for more
   details.

3.1.5.  pnfs_osd_data_map4

   struct pnfs_osd_data_map4 {
       length4                     stripe_unit;
       uint32_t                    group_width;
       uint32_t                    group_depth;
       uint32_t                    mirror_cnt;
       pnfs_osd_raid_algorithm4    raid_algorithm;
   };

   The pnfs_osd_data_map4 structure parameterizes the algorithm that
   maps a file's contents over the component objects.  Instead of
   limiting the system to simple striping scheme where loss of a single
   component object results in data loss, the map parameters support
   mirroring and more complicated schemes that protect against loss of a
   component object.

   The stripe_unit is the number of bytes placed on one component before
   advancing to the next one in the list of components.  The number of
   bytes in a full stripe is stripe_unit times the number of components.
   In some raid schemes, a stripe includes redundant information (i.e.,
   parity) that lets the system recover from loss or damage to a
   component object.

   The group_width and group_depth parameters allow a nested striping
   pattern.  If there is no nesting, then group_width and group_depth
   MUST be zero.  Otherwise, the group_width defines the width of a data
   stripe, and the group_depth defines how many stripes are accessed
   before advancing to the next group of components in the list of
   component objects for the file.  The size of the components array
   MUST be a multiple of group_width.

   The mirror_cnt is used to replicate a file by replicating its
   component objects.  If there is no mirroring, then mirror_cnt MUST be
   0.  If mirror_cnt is greater than zero, then the size of the

Halevy, et al.            Expires March 8, 2008                [Page 10]

Internet-Draft                pnfs objects                September 2007

   component array MUST be a multiple of (mirror_cnt+1).

   See Section 3.2 for more details.

3.2.  Data Mapping Schemes

   This section describes the different data mapping schemes in detail.

3.2.1.  Simple Striping

   The object layout always uses a "dense" layout as described in the
   pNFS document.  This means that the second stripe unit of the file
   starts at offset 0 of the second component, rather than at offset
   stripe_unit bytes.  After a full stripe has been written, the next
   stripe unit is appended to the first component object in the list
   without any holes in the component objects.  The mapping from the
   logical offset within a file (L) to do the component object C and
   object-specific offset O is defined by the following equations:

   L = logical offset into the file
   W = total number of components
   S = W * stripe_unit
   N = L / S
   C = (L-(N*S)) / stripe_unit
   O = (N*stripe_unit)+(L%stripe_unit)

   In these equations, S is the number of bytes in a full stripe, and N
   is the stripe number.  C is an index into the array of components, so
   it selects a particular object storage device.  Both N and C count
   from zero.  O is the offset within the object that corresponds to the
   file offset.  Note that this computation does not accommodate the
   same object appearing in the component array multiple times.

   For example, consider an object striped over four devices, <D0 D1 D2
   D3>.  The stripe_unit is 4096 bytes.  The stripe width S is thus 4 *
   4096 = 16384.

Halevy, et al.            Expires March 8, 2008                [Page 11]

Internet-Draft                pnfs objects                September 2007

   Offset 0:
     N = 0 / 16384 = 0
     C = 0-0/4096 = 0 (D0)
     O = 0*4096 + (0%4096) = 0

   Offset 4096:
     N = 4096 / 16384 = 0
     C = (4096-(0*16384)) / 4096 = 1 (D1)
     O = (0*4096)+(4096%4096) = 0

   Offset 9000:
     N = 9000 / 16384 = 0
     C = (9000-(0*16384)) / 4096 = 2 (D2)
     O = (0*4096)+(9000%4096) = 808

   Offset 132000:
     N = 132000 / 16384 = 8
     C = (132000-(8*16384)) / 4096 = 0
     O = (8*4096) + (132000%4096) = 33696

3.2.2.  Nested Striping

   The group_width and group_depth parameters allow a nested striping
   pattern.  If there is no nesting, then group_width and group_depth
   MUST be zero.  Otherwise, the group_width defines the width of a data
   stripe, and the group_depth defines how many stripes are written
   before advancing to the next group of components in the list of
   component objects for the file.  The size of the components array
   MUST be a multiple of group_width.  The math used to map from a file
   offset to a component object and offset within that object is shown
   below.  The computations map from the logical offset L to the
   component index C and offset relative O within that component object.

   L = logical offset into the file
   W = total number of components
   S = stripe_unit * group_depth * W
   T = stripe_unit * group_depth * group_width
   U = stripe_unit * group_width
   M = L / S
   G = (L - (M * S)) / T
   H = (L - (M * S)) % T
   N = H / U
   C = (H - (N * U)) / stripe_unit + G * group_width
   O = L % stripe_unit + N * stripe_unit + M * group_depth * stripe_unit

   In these equations, S is the number of bytes striped across all
   component objects before the pattern repeats.  T is the number of
   bytes striped within a group of component objects before advancing to

Halevy, et al.            Expires March 8, 2008                [Page 12]

Internet-Draft                pnfs objects                September 2007

   the next group.  U is the number of bytes in a stripe within a group.
   M is the "major" (i.e., across all components) stripe number, and N
   is the "minor" (i.e., across the group) stripe number.  G counts the
   groups from the beginning of the major stripe, and H is the byte
   offset within the group.

   For example, consider an object striped over 100 devices with a
   group_width of 10, a group_depth of 50, and a stripe_unit of 1 MB.
   In this scheme, 500 MB are written to the first 10 components, and
   5000 MB is written before the pattern wraps back around to the first
   component in the array.

   Offset 0:
     W = 100
     S = 1 MB * 50 * 100 = 5000 MB
     T = 1 MB * 50 * 10 = 500 MB
     U = 1 MB * 10 = 10 MB
     M = 0 / 5000 MB = 0
     G = (0 - (0 * 5000 MB)) / 500 MB = 0
     H = (0 - (0 * 5000 MB)) % 500 MB = 0
     N = 0 / 10 MB = 0
     C = (0 - (0 * 10 MB)) / 1 MB + 0 * 10 = 0
     O = 0 % 1 MB + 0 * 1 MB + 0 * 50 * 1 MB = 0

   Offset 27 MB:
     M = 27 MB / 5000 MB = 0
     G = (27 MB - (0 * 5000 MB)) / 500 MB = 0
     H = (27 MB - (0 * 5000 MB)) % 500 MB = 27 MB
     N = 27 MB / 10 MB = 2
     C = (27 MB - (2 * 10 MB)) / 1 MB + 0 * 10 = 7
     O = 27 MB % 1 MB + 2 * 1 MB + 0 * 50 * 1 MB = 2 MB

   Offset 7232 MB:
     M = 7232 MB / 5000 MB = 1
     G = (7232 MB - (1 * 5000 MB)) / 500 MB = 4
     H = (7232 MB - (1 * 5000 MB)) % 500 MB = 232 MB
     N = 232 MB / 10 MB = 23
     C = (232 MB - (23 * 10 MB)) / 1 MB + 4 * 10 = 42
     O = 7232 MB % 1 MB + 23 * 1 MB + 1 * 50 * 1 MB = 73 MB

3.2.3.  Mirroring

   The mirror_cnt is used to replicate a file by replicating its
   component objects.  If there is no mirroring, then mirror_cnt MUST be
   0.  If mirror_cnt is greater than zero, then the size of the
   component array MUST be a multiple of (mirror_cnt+1).  Thus, for a
   classic mirror on two objects, mirror_cnt is one.  If group_width is
   also non-zero, then the size MUST be a multiple of group_width *

Halevy, et al.            Expires March 8, 2008                [Page 13]

Internet-Draft                pnfs objects                September 2007

   (mirror_cnt+1).  Replicas are adjacent in the components array, and
   the value C produced by the above equations is not a direct index
   into the components array.  Instead, the following equations
   determine the replica component index RCi, where i ranges from 0 to
   mirror_cnt.

   C = component index for striping or two-level striping
   i ranges from 0 to mirror_cnt, inclusive
   RCi = C * (mirror_cnt+1) + i

3.3.  RAID Algorithms

   pnfs_osd_raid_algorithm4 determines the algorithm and placement of
   redundant data.  This section defines the different RAID algorithms.

3.3.1.  PNFS_OSD_RAID_0

   PNFS_OSD_RAID_0 means there is no parity data, so all bytes in the
   component objects are data bytes located by the above equations for C
   and O. If a component object is unavailable, the pNFS client can
   choose to return NULLs for the missing data, or it can retry the READ
   against the pNFS server, or it can return an EIO error.

3.3.2.  PNFS_OSD_RAID_4

   PNFS_OSD_RAID_4 means that the last component object, or the last in
   each group if group_width is > zero, contains parity information
   computed over the rest of the stripe with an XOR operation.  If a
   component object is unavailable, the client can read the rest of the
   stripe units in the damaged stripe and recompute the missing stripe
   unit by XORing the other stripe units in the stripe.  Or the client
   can replay the READ against the pNFS server which will presumably
   perform the reconstructed read on the client's behalf.

   When parity is present in the file, then there is an additional
   computation to map from the file offset L to the offset that accounts
   for embedded parity, L'.  First compute L', and then use L' in the
   above equations for C and O.

   L = file offset, not accounting for parity
   P = number of parity devices in each stripe
   W = group_width, if not zero, else size of component array
   N = L / (W-P * stripe_unit)
   L' = N * (W * stripe_unit) +
        (L % (W-P * stripe_unit))

Halevy, et al.            Expires March 8, 2008                [Page 14]

Internet-Draft                pnfs objects                September 2007

3.3.3.  PNFS_OSD_RAID_5

   PNFS_OSD_RAID_5 means that the position of the parity data is rotated
   on each stripe.  In the first stripe, the last component holds the
   parity.  In the second stripe, the next-to-last component holds the
   parity, and so on.  In this scheme, all stripe units are rotated so
   that I/O is evenly spread across objects as the file is read
   sequentially.  The rotated parity layout is illustrated here, with
   numbers indicating the stripe unit.

   0 1 2 P
   4 5 P 3
   8 P 6 7
   P 9 a b

   To compute the component object C, first compute the offset that
   accounts for parity L' and use that to compute C. Then rotate C to
   get C'.  Finally, increase C' by one if the parity information comes
   at or before C' within that stripe.  The following equations
   illustrate this by computing I, which is the index of the component
   that contains parity for a given stripe.

   L = file offset, not accounting for parity
   W = group_width, if not zero, else size of component array
   N = L / (W-1 * stripe_unit)
   (Compute L' as describe above)
   (Compute C based on L' as described above)
   C' = (C - (N%W)) % W
   I = W - (N%W) - 1
   if (C' <= I) {
     C'++
   }

3.3.4.  PNFS_OSD_RAID_PQ

   PNFS_OSD_RAID_PQ is a double-parity scheme that uses the Reed-Solomon
   P+Q encoding scheme.  In this layout, the last two component objects
   hold the P and Q data, respectively.  P is parity computed with XOR,
   and Q is a more complex equation that is not described here.  The
   equations given above for embedded parity can be used to map a file
   offset to the correct component object by setting the number of
   parity components to 2 instead of 1 for RAID4 or RAID5.  Clients may
   simply choose to read data through the metadata server if two
   components are missing or damaged.

   Issue: This scheme also has a RAID_4 like layout where the ECC blocks
   are stored on the same components on every stripe and a rotated,
   RAID-5 like layout where the stripe units are rotated.  Should we

Halevy, et al.            Expires March 8, 2008                [Page 15]

Internet-Draft                pnfs objects                September 2007

   make the following properties orthogonal: RAID_4 or RAID_5 (i.e.,
   non-rotated or rotated), and then have the number of parity
   components and the associated algorithm be the orthogonal parameter?

3.3.5.  RAID Usage and implementation notes

   RAID layouts with redundant data in their stripes require additional
   serialization of updates to ensure correct operation.  Otherwise, if
   two clients simultaneously write to the same logical range of an
   object, the result could include different data in the same ranges of
   mirrored tuples, or corrupt parity information.  It is the
   responsibility of the metadata server to enforce serialization
   requirements such as this.  For example, the metadata server may do
   so by not granting overlapping write layouts within mirrored objects.

4.  Object-Based Layout Update

   layoutupdate4 is used in the LAYOUTCOMMIT operation to convey updates
   to the layout and additional information to the metadata server.  It
   is defined in the NFSv4.1 draft [6] as follows:

   struct layoutupdate4 {
       layouttype4             lou_type;
       opaque                  lou_body<>;
   };

   The layoutupdate4 type is an opaque value at the generic pNFS client
   level.  If the lou_type layout type is LAYOUT4_OSD2_OBJECTS, then the
   lou_body opaque value is defined by the pnfs_osd_layoutupdate4 type.

4.1.  pnfs_osd_layoutupdate4

   struct pnfs_osd_layoutupdate4 {
       pnfs_osd_deltaspaceused4    delta_space_used;
       pnfs_osd_ioerr4             ioerr<>;
   };

   Object-Based pNFS clients are not allowed to modify the layout.
   "delta_space_used" is used to convey capacity usage information back
   to the metadata server and, in case OSD I/O operations failed,
   "ioerr" is used to report these errors to the metadata server.

Halevy, et al.            Expires March 8, 2008                [Page 16]

Internet-Draft                pnfs objects                September 2007

4.1.1.  pnfs_osd_deltaspaceused4

   union pnfs_osd_deltaspaceused4 switch (bool valid) {
       case TRUE:
           int64_t     delta;  /* Bytes consumed by write activity */
       case FALSE:
           void;
   };

   pnfs_osd_deltaspaceused4 is used to convey space utilization
   information at the time of LAYOUTCOMMIT.  For the file system to
   properly maintain capacity used information, it needs to track how
   much capacity was consumed by WRITE operations performed by the
   client.  In this protocol, the OSD returns the capacity consumed by a
   write, which can be different than the number of bytes written
   because of internal overhead like block-based allocation and indirect
   blocks, and the client reflects this back to the pNFS server so it
   can accurately track quota.  The pNFS server can choose to trust this
   information coming from the clients and therefore avoid querying the
   OSDs at the time of LAYOUTCOMMIT.  If the client is unable to obtain
   this information from the OSD, it simply returns invalid
   delta_space_used.

4.1.2.  pnfs_osd_errno4

   enum pnfs_osd_errno4 {
       PNFS_OSD_ERR_EIO            = 1,
       PNFS_OSD_ERR_NOT_FOUND      = 2,
       PNFS_OSD_ERR_NO_SPACE       = 3,
       PNFS_OSD_ERR_BAD_CRED       = 4,
       PNFS_OSD_ERR_NO_ACCESS      = 5,
       PNFS_OSD_ERR_UNREACHABLE    = 6,
       PNFS_OSD_ERR_RESOURCE       = 7
   };

   pnfs_osd_errno4 is used to represent error types when read/write
   errors are reported to the metadata server.  The error codes serve as
   hints to the metadata server that may help it in diagnosing the exact
   reason for the error and in repairing it.

   o  PNFS_OSD_ERR_EIO indicates the operation failed because the Object
      Storage Device experienced a failure trying to access the object.
      The most common source of these errors is media errors, but other
      internal errors might cause this.  In this case, the metadata
      server should go examine the broken object more closely, hence it
      should be used as the default error code.

Halevy, et al.            Expires March 8, 2008                [Page 17]

Internet-Draft                pnfs objects                September 2007

   o  PNFS_OSD_ERR_NOT_FOUND indicates the object ID specifies an object
      that does not exist on the Object Storage Device.

   o  PNFS_OSD_ERR_NO_SPACE indicates the operation failed because the
      Object Storage Device ran out of free capacity during the
      operation.

   o  PNFS_OSD_ERR_BAD_CRED indicates the security parameters are not
      valid.  The primary cause of this is that the capability has
      expired, or the access policy tag (a.k.a, capability version
      number) has been changed to revoke capabilities.  The client will
      need to return the layout and get a new one with fresh
      capabilities.

   o  PNFS_OSD_ERR_NO_ACCESS indicates the capability does not allow the
      requested operation.  This should not occur in normal operation
      because the metadata server should give out correct capabilities,
      or none at all.

   o  PNFS_OSD_ERR_UNREACHABLE indicates the client did not complete the
      I/O operation at the Object Storage Device due to a communication
      failure.  Whether the I/O operation was executed by the OSD or not
      is undetermined.

   o  PNFS_OSD_ERR_RESOURCE indicates the client did not issue the I/O
      operation due to a local problem on the initiator (i.e. client)
      side, e.g., when running out of memory.  The client MUST guarantee
      that the OSD command was never dispatched to the OSD.

4.1.3.  pnfs_osd_ioerr4

   struct pnfs_osd_ioerr4 {
       pnfs_osd_objid4     component;
       length4             comp_offset;
       length4             comp_length;
       bool                iswrite;
       pnfs_osd_errno4     errno;
   };

   The pnfs_osd_ioerr4 structure is used to return error indications for
   objects that generated errors during data transfers.  These are hints
   to the metadata server that there are problems with that object.  For
   each error, "component", "comp_offset", and "comp_length" represent
   the object and byte range within the component object in which the
   error occurred. "iswrite" is set to "true" if the failed OSD
   operation was data modifying, and "errno" represents the type of
   error.

Halevy, et al.            Expires March 8, 2008                [Page 18]

Internet-Draft                pnfs objects                September 2007

5.  Object-Based Creation Layout Hint

   The layouthint4 type is defined in the NFSv4.1 draft [6] as follows:

   struct layouthint4 {
       layouttype4           loh_type;
       opaque                loh_body<>;
   };

   The layouthint4 structure is used by the client to pass in a hint
   about the type of layout it would like created for a particular file.
   If the loh_type layout type is LAYOUT4_OSD2_OBJECTS, then the
   loh_body opaque value is defined by the pnfs_osd_layouthint4 type.

5.1.  pnfs_osd_layouthint4

   union num_comps_hint4 switch (bool valid) {
       case TRUE:
           uint32_t            num_comps;
       case FALSE:
           void;
   };

   union stripe_unit_hint4 switch (bool valid) {
       case TRUE:
           length4             stripe_unit;
       case FALSE:
           void;
   };

   union group_width_hint4 switch (bool valid) {
       case TRUE:
           uint32_t            group_width;
       case FALSE:
           void;
   };

   union group_depth_hint4 switch (bool valid) {
       case TRUE:
           uint32_t            group_depth;
       case FALSE:
           void;
   };

   union mirror_cnt_hint4 switch (bool valid) {
       case TRUE:
           uint32_t            mirror_cnt;
       case FALSE:

Halevy, et al.            Expires March 8, 2008                [Page 19]

Internet-Draft                pnfs objects                September 2007

           void;
   };

   union raid_algorithm_hint4 switch (bool valid) {
       case TRUE:
           pnfs_osd_raid_algorithm4    raid_algorithm;
       case FALSE:
           void;
   };

   struct pnfs_osd_layouthint4 {
       num_comps_hint4         num_comps_hint;
       stripe_unit_hint4       stripe_unit_hint;
       group_width_hint4       group_width_hint;
       group_depth_hint4       group_depth_hint;
       mirror_cnt_hint4        mirror_cnt_hint;
       raid_algorithm_hint4    raid_algorithm_hint;
   };

   This type conveys hints for the desired data map.  All parameters are
   optional so the client can give values for only the parameters it
   cares about, e.g. it can provide a hint for the desired number of
   mirrored components, regardless of the the raid algorithm selected
   for the file.  The server should make an attempt to honor the hints
   but it can ignore any or all of them at its own discretion and
   without failing the respective create operation.

   The num_comps hint can be used to limit the total number of component
   objects comprising the file.  All other hints correspond directly to
   the different fields of pnfs_osd_data_map4.

6.  Layout Segments

   The pnfs layout operations operate on logical byte ranges.  There is
   no requirement in the protocol for any relationship between byte
   ranges used in LAYOUTGET to acquire layouts and byte ranges used in
   CB_LAYOUTRECALL, LAYOUTCOMMIT, or LAYOUTRETURN.  However, using OSD
   capabilities poses limitations on these operations since the
   capabilities associated with layout segments cannot be merged or
   split.  The following guidelines should be followed for proper
   operation of object-based layouts.

6.1.  CB_LAYOUTRECALL and LAYOUTRETURN

   In general, the object-based layout driver should keep track of each
   layout segment it got, keeping record of the segment's iomode,
   offset, and length.  The server should allow the client to get

Halevy, et al.            Expires March 8, 2008                [Page 20]

Internet-Draft                pnfs objects                September 2007

   multiple overlapping layout segments but is free to recall the layout
   to prevent overlap.

   In response to CB_LAYOUTRECALL, the client should return all layout
   segments matching the given iomode and overlapping with the recalled
   range.  When returning the layouts for this byte range with
   LAYOUTRETURN the client MUST NOT return a sub-range of a layout
   segment it has; each LAYOUTRETURN sent MUST completely cover at least
   one outstanding layout segment.

   The server, in turn, should release any segment that exactly matches
   the clientid, iomode, and byte range given in LAYOUTRETURN.  If no
   exact match is found then the server should release all layout
   segments matching the clientid and iomode and that are fully
   contained in the returned byte range.  If none are found and the byte
   range is a subset of an outstanding layout segment with for the same
   clientid and iomode, then the client can be considered malfunctioning
   and the server SHOULD recall all layouts from this client to reset
   its state.  If this behavior repeats the server SHOULD deny all
   LAYOUTGETs from this client.

6.2.  LAYOUTCOMMIT

   LAYOUTCOMMIT is only used by object-based pNFS to convey modified
   attributes hints and/or to report I/O errors to the MDS.  Therefore,
   the offset and length in LAYOUTCOMMIT4args are reserved for future
   use and should be set to 0.  However, component byte ranges in the
   optional pnfs_osd_ioerr4 structure are used for recovering the object
   and MUST be set by the client to cover all failed I/O operations to
   the component.

7.  Recalling Layouts

   The object-based metadata server should recall outstanding layouts in
   the following cases:

   o  When the file's security policy changes, i.e.  ACLs or permission
      mode bits are set.

   o  When the file's aggregation map changes, rendering outstanding
      layouts invalid.

   o  When there are sharing conflicts.  For example, the server will
      issue stripe aligned layout segments for RAID-5 objects.  To
      prevent corruption of the file's parity, Multiple clients must not
      hold valid write layouts for the same stripes.  An outstanding RW
      layout should be recalled when a conflicting LAYOUTGET is received

Halevy, et al.            Expires March 8, 2008                [Page 21]

Internet-Draft                pnfs objects                September 2007

      from a different client for LAYOUTIOMODE4_RW and for a byte-range
      overlapping with the outstanding layout segment.

7.1.  CB_RECALL_ANY

   The metadata server can use the CB_RECALL_ANY callback operation to
   notify the client to return some or all of its layouts.  The NFSv4.1
   draft [6] defines the following types:

   const RCA4_TYPE_MASK_OBJ_LAYOUT_MIN     = 8;
   const RCA4_TYPE_MASK_OBJ_LAYOUT_MAX     = 11;

   struct  CB_RECALL_ANY4args      {
       uint32_t        craa_objects_to_keep;
       bitmap4         craa_type_mask;
   };

   Typically, CB_RECALL_ANY will be used to recall client state when the
   server needs to reclaim resources.  The craa_type_mask bitmap
   specifies the type of resources that are recalled and the
   craa_objects_to_keep value specifies how many of the recalled objects
   the client is allowed to keep.  The object-based layout type mask
   flags are defined as follows.  They represent the iomode of the
   recalled layouts.  In response, the client SHOULD return layouts of
   the recalled iomode that it needs the least, keeping at most
   craa_objects_to_keep object-based layouts.

  const PNFS_OSD_RCA4_TYPE_MASK_READ  = RCA4_TYPE_MASK_OBJ_LAYOUT_MIN;
  const PNFS_OSD_RCA4_TYPE_MASK_RW    = RCA4_TYPE_MASK_OBJ_LAYOUT_MIN+1;
  const PNFS_OSD_RCA4_TYPE_MASK_ANY   = RCA4_TYPE_MASK_OBJ_LAYOUT_MIN+2;

   The PNFS_OSD_RCA4_TYPE_MASK_READ flag notifies the client to return
   layouts of iomode LAYOUTIOMODE4_READ.  Similarly, the
   PNFS_OSD_RCA4_TYPE_MASK_RW flag notifies the client to return layouts
   of iomode LAYOUTIOMODE4_RW.  The PNFS_OSD_RCA4_TYPE_MASK_ANY flag
   notifies the client to return layouts of either iomode.

8.  Client Fencing

   In cases where clients are uncommunicative and their lease has
   expired or when clients fail to return recalled layouts in a timely
   manner the server MAY revoke client layouts and/or device address
   mappings and reassign these resources to other clients.  To avoid
   data corruption, the metadata server MUST fence off the revoked
   clients from the respective objects as described in Section 9.4.

Halevy, et al.            Expires March 8, 2008                [Page 22]

Internet-Draft                pnfs objects                September 2007

9.  Security Considerations

   The pNFS extension partitions the NFSv4 file system protocol into two
   parts, the control path and the data path (storage protocol).  The
   control path contains all the new operations described by this
   extension; all existing NFSv4 security mechanisms and features apply
   to the control path.  The combination of components in a pNFS system
   is required to preserve the security properties of NFSv4 with respect
   to an entity accessing data via a client, including security
   countermeasures to defend against threats that NFSv4 provides
   defenses for in environments where these threats are considered
   significant.

   The metadata server enforces the file access-control policy at
   LAYOUTGET time.  The client should use suitable authorization
   credentials for getting the layout for the requested iomode (READ or
   RW) and the server verifies the permissions and ACL for these
   credentials, possibly returning NFS4ERR_ACCESS if the client is not
   allowed the requested iomode.  If the LAYOUTGET operation succeeds
   the client receives, as part of the layout, a set of object
   capabilities allowing it I/O access to the specified objects
   corresponding to the requested iomode.  When the client acts on I/O
   operations on behalf of its local users it MUST authenticate and
   authorize the user by issuing respective OPEN and ACCESS calls to the
   metadata server, similarly to having NFSv4 data delegations.  If
   access is allowed the client uses the corresponding (READ or RW)
   capabilities to perform the I/O operations at the object-storage
   devices.  When the metadata server receives a request to change
   file's permissions or ACL it SHOULD recall all layouts for that file
   and it MUST change the capability version attribute on all objects
   comprising the file to implicitly invalidate any outstanding
   capabilities before committing to the new permissions and ACL.  Doing
   this will ensure that clients re-authorize their layouts according to
   the modified permissions and ACL by requesting new layouts.
   Recalling the layouts in this case is courtesy of the server intended
   to prevent clients from getting an error on I/Os done after the
   capability version changed.

   The object storage protocol MUST implement the security aspects
   described in version 1 of the T10 OSD protocol definition [2].  The
   standard defines four security methods: NOSEC, CAPKEY, CMDRSP, and
   ALLDATA.  To provide minimum level of security allowing verification
   and enforcement of the server access control policy using the layout
   security credentials, the NOSEC security method MUST NOT be used for
   I/O operation.  It MAY only be used to get the System ID attribute
   when the metadata server provided only the OSD name with the device
   address.  The remainder of this section gives an overview of the
   security mechanism described in that standard.  The goal is to give

Halevy, et al.            Expires March 8, 2008                [Page 23]

Internet-Draft                pnfs objects                September 2007

   the reader a basic understanding of the object security model.  Any
   discrepancies between this text and the actual standard are obviously
   to be resolved in favor of the OSD standard.

9.1.  OSD Security Data Types

   There are three main data types associated with object security: a
   capability, a credential, and security parameters.  The capability is
   a set of fields that specifies an object and what operations can be
   performed on it.  A credential is a signed capability.  Only a
   security manager that knows the secret device keys can correctly sign
   a capability to form a valid credential.  In pNFS, the file server
   acts as the security manager and returns signed capabilities (i.e.,
   credentials) to the pNFS client.  The security parameters are values
   computed by the issuer of OSD commands (i.e., the client) that prove
   they hold valid credentials.  The client uses the credential as a
   signing key to sign the requests it makes to OSD, and puts the
   resulting signatures into the security_parameters field of the OSD
   command.  The object storage device uses the secret keys it shares
   with the security manager to validate the signature values in the
   security parameters.

   The security types are opaque to the generic layers of the pNFS
   client.  The credential contents are defined as opaque within the
   pnfs_osd_object_cred4 type.  Instead of repeating the definitions
   here, the reader is referred to section 4.9.2.2 of the OSD standard.

9.2.  The OSD Security Protocol

   The object storage protocol relies on a cryptographically secure
   capability to control accesses at the object storage devices.
   Capabilities are generated by the metadata server, returned to the
   client, and used by the client as described below to authenticate
   their requests to the Object Storage Device (OSD).  Capabilities
   therefore achieve the required access and open mode checking.  They
   allow the file server to define and check a policy (e.g., open mode)
   and the OSD to enforce that policy without knowing the details (e.g.,
   user IDs and ACLs).

   Since capabilities are tied to layouts, and since they are used to
   enforce access control, when the file ACL or mode changes the
   outstanding capabilities MUST be revoked to enforce the new access
   permissions.  The server SHOULD recall layouts to allow clients to
   gracefully return their capabilities before the access permissions
   change.

   Each capability is specific to a particular object, an operation on
   that object, a byte range w/in the object (in OSDv2), and has an

Halevy, et al.            Expires March 8, 2008                [Page 24]

Internet-Draft                pnfs objects                September 2007

   explicit expiration time.  The capabilities are signed with a secret
   key that is shared by the object storage devices (OSD) and the
   metadata managers.  Clients do not have device keys so they are
   unable to forge the signatures in the security parameters.  The
   combination of a capability, the OSD system id, and a signature is
   called a "credential" in the OSD specification.

   The details of the security and privacy model for Object Storage are
   defined in the T10 OSD standard.  The following sketch of the
   algorithm should help the reader understand the basic model.

   LAYOUTGET returns a CapKey and a Cap which, together with the OSD
   SystemID, are also called a credential.  It is a capability and a
   signature over that capability and the SystemID.  The OSD Standard
   refers to the CapKey as the "Credential integrity check value" and to
   the ReqMAC as the "Request integrity check value".

   CapKey = MAC<SecretKey>(Cap, SystemID)
   Credential = {Cap, SystemID, CapKey}

   The client uses CapKey to sign all the requests it issues for that
   object using the respective Cap. In other words, the Cap appears in
   the request to the storage device, and that request is signed with
   the CapKey as follows:

   ReqMAC = MAC<CapKey>(Req, ReqNonce)
   Request = {Cap, Req, ReqNonce, ReqMAC}

   The following is sent to the OSD: {Cap, Req, ReqNonce, ReqMAC}.  The
   OSD uses the SecretKey it shares with the metadata server to compare
   the ReqMAC the client sent with a locally computed value:

   LocalCapKey = MAC<SecretKey>(Cap, SystemID)
   LocalReqMAC = MAC<LocalCapKey>(Req, ReqNonce)

   and if they match the OSD assumes that the capabilities came from an
   authentic metadata server and allows access to the object, as allowed
   by the Cap.

9.3.  Protocol Privacy Requirements

   Note that if the server LAYOUTGET reply, holding CapKey and Cap, is
   snooped by another client, it can be used to generate valid OSD
   requests (within the Cap access restrictions).

   To provide the required privacy requirements for the capability key
   returned by LAYOUTGET, the GSS-API can be used, e.g. by using the
   RPCSEC_GSS privacy method to send the LAYOUTGET operation or by using

Halevy, et al.            Expires March 8, 2008                [Page 25]

Internet-Draft                pnfs objects                September 2007

   the SSV key to encrypt the capability_key using the GSS_Wrap()
   function.  Two general ways to provide privacy in the absence of GSS-
   API that are independent of NFSv4 are either an isolated network such
   as a VLAN or a secure channel provided by IPsec [8].

9.4.  Revoking Capabilities

   At any time, the metadata server may invalidate all outstanding
   capabilities on an object by changing its POLICY ACCESS TAG
   attribute.  The value of the POLICY ACCESS TAG is part of a
   capability, and it must match the state of the object attribute.  If
   they do not match, the OSD rejects accesses to the object with the
   sense key set to ILLEGAL REQUEST and an additional sense code set to
   INVALID FIELD IN CDB.  When a client attempts to use a capability and
   is rejected this way, it should issue a LAYOUTCOMMIT for the object
   and specify PNFS_OSD_BAD_CRED in the ioerr parameter.  The client may
   elect to issue a compound LAYOUTRETURN/LAYOUTGET (or LAYOUTCOMMIT/
   LAYOUTRETURN/LAYOUTGET) to attempt to fetch a refreshed set of
   capabilities.

   The metadata server may elect to change the access policy tag on an
   object at any time, for any reason (with the understanding that there
   is likely an associated performance penalty, especially if there are
   outstanding layouts for this object).  The metadata server MUST
   revoke outstanding capabilities when any one of the following occurs:

   o  the permissions on the object change,

   o  a conflicting mandatory byte-range lock is granted, or

   o  a layout is revoked and reassigned to another client

   A pNFS client will typically hold one layout for each byte range for
   either READ or READ/WRITE.  The client's credentials are checked by
   the metadata server at LAYOUTGET time and it is the client's
   responsibility to enforce access control among multiple users
   accessing the same file.  It is neither required nor expected that
   the pNFS client will obtain a separate layout for each user accessing
   a shared object.  The client SHOULD use OPEN and ACCESS calls to
   check user permissions when performing I/O so that the server's
   access control policies are correctly enforced.  The result of the
   ACCESS operation may be cached while the client holds a valid layout
   as the server is expected to recall layouts when the file's access
   permissions or ACL change.

Halevy, et al.            Expires March 8, 2008                [Page 26]

Internet-Draft                pnfs objects                September 2007

10.  IANA Considerations

   As described in the NFSv4.1 draft [6], new layout type numbers will
   be requested from IANA.  This document defines the protocol
   associated with the existing layout type number,
   LAYOUT4_OSD2_OBJECTS, and it requires no further actions for IANA.

11.  References

11.1.  Normative References

   [1]  Bradner, S., "Key words for use in RFCs to Indicate Requirement
        Levels", RFC 2119, March 1997.

   [2]  Weber, R., "SCSI Object-Based Storage Device Commands",
        July 2004, <http://www.t10.org/ftp/t10/drafts/osd/osd-r10.pdf>.

   [3]  Linn, J., "Generic Security Service Application Program
        Interface Version 2, Update 1", RFC 2743, January 2000.

   [4]  Eisler, M., "XDR: External Data Representation Standard",
        STD 67, RFC 4506, May 2006.

   [5]  Shepler, S., Callaghan, B., Robinson, D., Thurlow, R., Beame,
        C., Eisler, M., and D. Noveck, "Network File System (NFS)
        version 4 Protocol", RFC 3530, April 2003.

11.2.  Informative References

   [6]  Shepler, S., Eisler, M., and D. Noveck, "NFSv4 Minor Version 1",
        March 2007, <http://www.ietf.org/internet-drafts/
        draft-ietf-nfsv4-minorversion1-13.txt>.

   [7]  Weber, R., "SCSI Object-Based Storage Device Commands -2
        (OSD-2)", January 2007,
        <http://www.t10.org/ftp/t10/drafts/osd2/osd2r02.pdf>.

   [8]  Kent, S. and K. Seo, "Security Architecture for the Internet
        Protocol", RFC 4301, December 2005.

Appendix A.  Acknowledgments

   Todd Pisek was a co-editor of the initial drafts for this document.

Halevy, et al.            Expires March 8, 2008                [Page 27]

Internet-Draft                pnfs objects                September 2007

Authors' Addresses

   Benny Halevy
   Panasas, Inc.
   1501 Reedsdale St. Suite 400
   Pittsburgh, PA  15233
   USA

   Phone: +1-412-323-3500
   Email: [email-masked]
   URI:   http://www.panasas.com/

   Brent Welch
   Panasas, Inc.
   6520 Kaiser Drive
   Fremont, CA  95444
   USA

   Phone: +1-650-608-7770
   Email: [email-masked]
   URI:   http://www.panasas.com/

   Jim Zelenka
   Panasas, Inc.
   1501 Reedsdale St. Suite 400
   Pittsburgh, PA  15233
   USA

   Phone: +1-412-323-3500
   Email: [email-masked]
   URI:   http://www.panasas.com/

Halevy, et al.            Expires March 8, 2008                [Page 28]

Internet-Draft                pnfs objects                September 2007

Full Copyright Statement

   Copyright (C) The IETF Trust (2007).

   This document is subject to the rights, licenses and restrictions
   contained in BCP 78, and except as set forth therein, the authors
   retain all their rights.

   This document and the information contained herein are provided on an
   "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
   OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
   THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
   OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
   THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
   WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Intellectual Property

   The IETF takes no position regarding the validity or scope of any
   Intellectual Property Rights or other rights that might be claimed to
   pertain to the implementation or use of the technology described in
   this document or the extent to which any license under such rights
   might or might not be available; nor does it represent that it has
   made any independent effort to identify any such rights.  Information
   on the procedures with respect to rights in RFC documents can be
   found in BCP 78 and BCP 79.

   Copies of IPR disclosures made to the IETF Secretariat and any
   assurances of licenses to be made available, or the result of an
   attempt made to obtain a general license or permission for the use of
   such proprietary rights by implementers or users of this
   specification can be obtained from the IETF on-line IPR repository at
   http://www.ietf.org/ipr.

   The IETF invites any interested party to bring to its attention any
   copyrights, patents or patent applications, or other proprietary
   rights that may cover technology that may be required to implement
   this standard.  Please address the information to the IETF at
   [email-masked].

Acknowledgment

   Funding for the RFC Editor function is provided by the IETF
   Administrative Support Activity (IASA).

Halevy, et al.            Expires March 8, 2008                [Page 29]


<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE rfc SYSTEM "xml/rfc2629.dtd">
<?xml-stylesheet type='text/xsl' href='xml/rfc2629.xslt' ?>

<!-- XML source for the pnfs over objects internet draft document -->

<!-- To generate text with the xml2rfc tool tclsh8.3 xml2rfc.tcl 
     xml2rfc this_file.xml that_file.txt which puts the formatted 
     text into that_file.txt -->

<!-- processing instructions (for a complete list and description,
     see file http://xml.resource.org/authoring/README.html -->

<!-- try to enforce the ID-nits conventions and DTD validity -->

<?rfc strict="yes" ?>

<!-- items used when reviewing the document -->

<?rfc comments="no" ?>  <!-- controls display of <cref> elements -->
<?rfc inline="no" ?>    <!-- when no, put comments at end in comments section,
                                otherwise, put inline -->
<?rfc editing="no" ?>   <!-- when yes, insert editing marks -->

<!-- create table of contents (set it options).  
     Note the table of contents may be omitted
     for very short documents --> 

<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="3"?>

<!-- choose the options for the references. Some like
     symbolic tags in the references (and citations)
     and others prefer numbers. --> 

<?rfc symrefs="no"?>
<?rfc sortrefs="yes" ?>

<!-- these two save paper: start new paragraphs from the same page etc. -->

<?rfc compact="yes" ?>
<?rfc subcompact="no" ?>

<!-- end of list of processing instructions -->

<rfc
    category="std"
    ipr="full3978"
    docName="draft-ietf-nfsv4-pnfs-obj-04">

<front>
    <title abbrev="pnfs objects">Object-based pNFS Operations</title>

    <author fullname="Benny Halevy" 
            initials="B." 
            surname="Halevy">
        <organization abbrev="Panasas">Panasas, Inc.</organization>
        <address>
            <postal>
                <street>1501 Reedsdale St. Suite 400</street>
                <city>Pittsburgh</city>
                <region>PA</region>
                <code>15233</code>
                <country>USA</country>
            </postal>
            <phone>+1-412-323-3500</phone>
            <email>[email-masked]</email>
            <uri>http://www.panasas.com/</uri>
        </address>
    </author>

    <author fullname="Brent Welch" 
            initials="B." 
            surname="Welch">
        <organization abbrev="Panasas">Panasas, Inc.</organization>
        <address>
            <postal>
                <street>6520 Kaiser Drive</street>
                <city>Fremont</city>
                <region>CA</region>
                <code>95444</code>
                <country>USA</country>
            </postal>
            <phone>+1-650-608-7770</phone>
            <email>[email-masked]</email>
            <uri>http://www.panasas.com/</uri>
        </address>
    </author>

    <author fullname="Jim Zelenka" 
            initials="J." 
            surname="Zelenka">
        <organization abbrev="Panasas">Panasas, Inc.</organization>
        <address>
            <postal>
                <street>1501 Reedsdale St. Suite 400</street>
                <city>Pittsburgh</city>
                <region>PA</region>
                <code>15233</code>
                <country>USA</country>
            </postal>
            <phone>+1-412-323-3500</phone>
            <email>[email-masked]</email>
            <uri>http://www.panasas.com/</uri>
        </address>
    </author>

    <date year="2007" month="September"/>

    <area>Transport</area>
    <workgroup>NFSv4</workgroup>

    <abstract>
      <t>
      This Internet-Draft provides a description of the object-based
      pNFS extension for NFSv4.  This is a companion to the main
      pnfs specification in the NFSv4 Minor Version 1 Internet Draft,
      which is currently draft-ietf-nfsv4-minorversion1-13.txt.
      </t>
    </abstract>

    <note title="Requirements Language">
<t>The key words &quot;MUST&quot;, &quot;MUST NOT&quot;,
&quot;REQUIRED&quot;, &quot;SHALL&quot;, &quot;SHALL NOT&quot;,
&quot;SHOULD&quot;, &quot;SHOULD NOT&quot;, &quot;RECOMMENDED&quot;,
&quot;MAY&quot;, and &quot;OPTIONAL&quot; in this document are to be
interpreted as described in <xref target="RFC2119">RFC 2119</xref>.
</t>

    </note>

</front>

<middle>

<section title="Introduction">
 <t>
In pNFS, the file server returns typed layout structures that describe where
file data is located.  There are different layouts for different storage systems
and methods of arranging data on storage devices.  This document describes
the layouts used 
with object-based storage devices (OSD) that are accessed
according to the iSCSI/OSD storage protocol standard
(<xref target="osd standard">SNIA T10/1355-D</xref>).
 </t>
 <t>
An "object" is a container for data and attributes, and files are stored in
one or more objects.  The OSD protocol specifies several operations on
objects, including READ, WRITE, FLUSH, GET ATTRIBUTES, SET ATTRIBUTES, CREATE and DELETE.
However, using the object-based layout the client only uses the READ, WRITE, GET ATTRIBUTES and FLUSH
commands. The other commands are only used by the pNFS server.
 </t>
 <t>
An object-based layout for pNFS includes object identifiers, capabilities
that allow clients to READ or WRITE those objects, and various parameters
that control how file data is striped across their component objects.
The OSD protocol 
has a capability-based security scheme that allows the
pNFS server to control what operations and what objects can be used by clients.
This scheme is described in more detail in the 
<xref target="Security Considerations">Security Considerations section</xref>.
 </t>
</section>

<section title="Object Storage Device Addressing and Discovery">
 <t>
   Data operations to an OSD require the client to know the "address"
   of each OSD's root object. The root object is synonymous with SCSI logical unit.
   The client specifies SCSI logical units to its SCSI stack using a
   representation local to the client. Because these representations
   are local, GETDEVICEINFO must return information that can be used
   by the client to select the correct local representation.
 </t>
 <t>
   In the block world, a set offset (logical block number or track/sector)
   contains a disk label. This label identifies the disk uniquely.
   In contrast, an OSD has a standard set of attributes on its root object.
   For device identification purposes
   the OSD System ID (root information attribute number 3) and/or
   OSD Name (root information attribute number 9) are used as the label.
   These appear in the pnfs_osd_deviceaddr4 type below under the
   "systemid" and "osdname" fields.
 </t>
 <t>
   In some situations, SCSI target discovery may need to be driven based
   on information contained in the GETDEVICEINFO response. One example of
   this is iSCSI targets that are not known to the client until a layout
   has been requested. Eventually iSCSI will adopt ANSI T10 SAM-3, at
   which time the World Wide Name (WWN aka, EUI-64/EUI-128) naming
   conventions can be specified. In addition, Fibre Channel (FC) SCSI
   targets have a unique WWN. Although these FC targets have already been
   discovered, some implementations may want to specify the WWN in
   addition to the label.  This information appears as the "target" and "lun"
   fields in the pnfs_osd_deviceaddr4 type described below.
 </t>
 <t>
   The systemid is used by the client, along with the object credential to sign
   each request with the request integrity check value.
   This method protects the client from unintentionally accessing a device if
   the device address mapping was changed (or revoked).
   The server computes the capability_key using its own view of the systemid
   associated with the respective deviceid present in the credential.  If the
   client's view of the deviceid mapping is stale, the client will use the wrong
   systemid (which must be system-wide unique) and the I/O request to the OSD will
   fail to pass the integrity check verification.
 </t>
 <t>
   To recover from this condition the client should report the error via LAYOUTCOMMIT,
   return the layout using LAYOUTRETURN, and invalidate all the device address mappings
   associated with this layout.
   The client can then ask for a new layout if it wishes using LAYOUTGET and
   resolve the referenced deviceids using GETDEVICEINFO or GETDEVICELIST.
 </t>
 <t>
 The server MUST provide either the systemid, the OSD name, or both.
 When the OSD name is present the client SHOULD get the root information
 attributes whenever it establishes communication with the OSD and verify
 that the OSD name it got from the OSD matches the one sent by the metadata server.
 If the systemid was not given by the server it MUST be taken from the OSD-provided attribute;
 note that in this case the OSD GET ATTRIBUTES operation must be performed with the NOSEC
 security method.
 </t>
 <section anchor="pnfs_osd_addr_type4" title="pnfs_osd_addr_type4">
  <t>
   The following enum specifies the manner in which a scsi target can
   be specified. The target can be specified as a network address,
   as an Internet Qualified Name (IQN), or by the World-Wide Name (WWN)
   of the target.
  </t>
  <figure>
   <artwork>
enum pnfs_obj_addr_type4 {
    OBJ_TARGET_NETADDR  = 1,
    OBJ_TARGET_IQN      = 2,
    OBJ_TARGET_WWN      = 3
};
   </artwork>
  </figure>
 </section>

 <section anchor="pnfs_osd_deviceaddr4" title="pnfs_osd_deviceaddr4">
  <t>
The specification for an object device address is as follows:
  </t>
  <figure>
   <artwork>
struct pnfs_osd_deviceaddr4 {
    union target switch (pnfs_osd_addr_type4 type) {
        case OBJ_TARGET_NETADDR:
            pnfs_netaddr4   netaddr;

        case OBJ_TARGET_IQN:
            string          iqn&lt;&gt;;

        case OBJ_TARGET_WWN:
            string          wwn&lt;&gt;;

        default:
            void;
    };
    uint64_t            lun;
    opaque              systemid&lt;&gt;;
    opaque              osdname&lt;&gt;;
};
   </artwork>
  </figure>
 </section>
</section>

<section title="Object-Based Layout">
 <t>
The layout4 type is defined in the
<xref target="NFSv4.1">NFSv4.1 draft</xref> as follows:
 </t>

 <figure>
  <artwork>
enum layouttype4 {
    LAYOUT4_NFSV4_1_FILES   = 1,
    LAYOUT4_OSD2_OBJECTS    = 2,
    LAYOUT4_BLOCK_VOLUME    = 3
};

struct layout_content4 {
    layouttype4             loc_type;
    opaque                  loc_body&lt;&gt;;
};

struct layout4 {
    offset4                 lo_offset;         
    length4                 lo_length;         
    layoutiomode4           lo_iomode;
    layout_content4         lo_content;
};

  </artwork>
 </figure>

 <t>
This document defines structure associated with
the layouttype4 value, LAYOUT4_OSD2_OBJECTS.
The <xref target="NFSv4.1">NFSv4.1 draft</xref> specifies the loc_body structure as an XDR type "opaque".
The opaque layout is uninterpreted by the generic pNFS client layers, but obviously
must be interpreted by the object-storage layout driver.  This document defines the
structure of this opaque value, pnfs_osd_layout4.
 </t>

 <section anchor="pnfs_osd_layout4" title="pnfs_osd_layout4">
  <figure>
   <artwork>
struct pnfs_osd_layout4 {
    pnfs_osd_data_map4      map;
    pnfs_osd_object_cred4   components&lt;&gt;;
};
   </artwork>
  </figure>
  <t>
The pnfs_osd_layout4 structure specifies a layout over a set of 
component objects.
The components field is an array of object identifiers
and security credentials that grant
access to each object.
The organization of the data is defined by the pnfs_osd_data_map4 type
that specifies how the file's data is mapped onto the component objects
(i.e., the striping pattern).
The data placement algorithm that maps file data
onto component objects assume that each component object
occurs exactly once in the array of components.
Therefore, component objects MUST appear in the component array
only once.
  </t>

  <t>
Note that the layout depends on the file size, which the client learns
from the generic return parameters of LAYOUTGET,
by doing GETATTR commands to the metadata server.
The client uses the file size
to decide if it
should fill holes with zeros, or return a short read.
Striping patterns can cause cases where component objects are shorter
than other components because a hole happens to correspond to the
last part of the component object.
  </t>

  <section anchor="pnfs_osd_objid4" title="pnfs_osd_objid4">
   <t>
An object is identified by a number, somewhat like an inode number.
The object storage model has a two level scheme, where the objects
within an object storage device are grouped into partitions.
   </t>
   <figure>
    <artwork>
struct pnfs_osd_objid4 {
    deviceid4       device_id;
    uint64_t        partition_id;
    uint64_t        object_id;
};
    </artwork>
   </figure>
   <t>
The pnfs_osd_objid4 type is used to identify an object within a partition
on a specified object storage device.
"device_id" selects the
object storage device from the set of available storage devices.
The device is identified with the deviceid4 type, which
is an index into addressing information about that device
returned by the GETDEVICELIST and GETDEVICEINFO pnfs operations.
Within an OSD, a partition is identified with a 64-bit number, "partition_id".
Within a partition, an object is identified with a 64-bit number, "object_id".
Creation and management of partitions
is outside the scope of this standard, and is a facility provided
by the object storage file system.
   </t>
  </section>

  <section anchor="pnfs_osd_version4" title="pnfs_osd_version4">
   <figure>
    <artwork>
enum pnfs_osd_version4 {
    PNFS_OSD_MISSING    = 0,
    PNFS_OSD_VERSION_1  = 1,
    PNFS_OSD_VERSION_2  = 2
};
    </artwork>
   </figure>
   <t>
The osd_version is used to indicate the OSD protocol version or whether
an object is missing (i.e., unavailable).  Some layout schemes
encode redundant information and can compensate for missing
components, but the data placement algorithm needs to know
what parts are missing.
   </t>

   <t>
At this time the OSD standard is at version 1.0, and
we anticipate a version 2.0 of the standard
((<xref target="OSD2">SNIA T10/1729-D</xref>)).
The second generation OSD protocol has additional proposed 
features to support more
robust error recovery, snapshots, and byte-range capabilities.
Therefore, the OSD version is explicitly called out
in the information returned in the layout.
(This information can also be deduced by looking inside
the capability type at the format field, which is the first byte.
The format value is 0x1 for an OSD v1 capability.
However, it seems most robust to call out the version explicitly.)
   </t>
  </section>

  <section anchor="pnfs_osd_object_cred4" title="pnfs_osd_object_cred4">
   <figure>
    <artwork>
enum pnfs_osd_cap_key_sec4 {
    PNFS_OSD_CAP_KEY_SEC_NONE = 0,
    PNFS_OSD_CAP_KEY_SEC_SSV  = 1,
};

struct pnfs_osd_object_cred4 {
    pnfs_osd_objid4         object_id;
    pnfs_osd_version4       osd_version;
    pnfs_osd_cap_key_sec4   cap_key_sec;
    opaque                  capability_key&lt;&gt;;
    opaque                  capability&lt;&gt;;
};
    </artwork>
   </figure>
   <t>
The pnfs_osd_object_cred4 structure is used to identify each component
comprising the file.
The object_id identifies the component object, the osd_version represents
the osd protocol version, or whether that component is unavailable, and
the capability and capability key,
along with the systemid from the pnfs_osd_deviceaddr,
provide the OSD security credentials needed
to access that object. The cap_key_sec value denotes the method used to secure
the capability_key
(see <xref target="OSD Security Data Types" /> for more details).
   </t>
   <t>
To comply with the OSD security requirements the capability key SHOULD
be transferred securely to prevent eavesdropping (see <xref target="Security Considerations" />).
Therefore, a client SHOULD either issue the LAYOUTGET operation via RPCSEC_GSS
with the privacy service or to previously establish an SSV for the sessions
via the NFSv4.1 SET_SSV operation.
The pnfs_osd_cap_key_sec4 type is used to identify the method used by the
server to secure the capability key.
   <list style='symbols'>
    <t>
PNFS_OSD_CAP_KEY_SEC_NONE denotes that the capability_key is not encrypted
in which case the client SHOULD issue the LAYOUTGET operation with
RPCSEC_GSS with the privacy service or the NFSv4.1 transport should
be secured by using methods that are external to NFSv4.1 like the use of
<xref target='RFC4301'>IPSEC</xref> for transporting the NFSV4.1 protocol.
    </t>
    <t>
PNFS_OSD_CAP_KEY_SEC_SSV denotes that the capability_key contents are
encrypted using the SSV GSS context and the capability key as
inputs to the GSS_Wrap() function (see <xref target='RFC2743' />) with
the conf_req_flag set to TRUE.  The client MUST use the secret SSV key
as part of the client's GSS context to decrypt the capability key
using the value of the capability_key field as the input_message to the
GSS_unwrap() function. Note that to prevent eavesdropping of the SSV key
the client SHOULD issue SET_SSV via RPCSEC_GSS with the privacy service.
    </t>
   </list>
   </t>
   <t>
The actual method chosen depends on whether the client established
a SSV key with the server and whether it issued the LAYOUTGET
operation with the RPCSEC_GSS privacy method.
Naturally, if the client did not establish a SSV key via SET_SSV
the server MUST use the PNFS_OSD_CAP_KEY_SEC_NONE method.
Otherwise, if the LAYOUTGET operation was not issued with the RPCSEC_GSS
privacy method the server SHOULD secure the capability_key with the
PNFS_OSD_CAP_KEY_SEC_SSV method.  The server MAY use the
PNFS_OSD_CAP_KEY_SEC_SSV method also when the LAYOUTGET operation
was issued with the RPCSEC_GSS privacy method.
   </t>
  </section>

  <section anchor="pnfs_osd_raid_algorithm4" title="pnfs_osd_raid_algorithm4">
   <figure>
    <artwork>
enum pnfs_osd_raid_algorithm4 {
    PNFS_OSD_RAID_0     = 1,
    PNFS_OSD_RAID_4     = 2,
    PNFS_OSD_RAID_5     = 3,
    PNFS_OSD_RAID_PQ    = 4     /* Reed-Solomon P+Q */
};
    </artwork>
   </figure>
   <t>
pnfs_osd_raid_algorithm4 represents the data redundancy
algorithm used to protect the file's contents.
See <xref target="RAID Algorithms" /> for more details.
   </t>
  </section>

  <section anchor="pnfs_osd_data_map4" title="pnfs_osd_data_map4">
   <figure>
    <artwork>
struct pnfs_osd_data_map4 {
    length4                     stripe_unit;
    uint32_t                    group_width;
    uint32_t                    group_depth;
    uint32_t                    mirror_cnt;
    pnfs_osd_raid_algorithm4    raid_algorithm;
};
    </artwork>
   </figure>
   <t>
The pnfs_osd_data_map4 structure parameterizes the algorithm that maps
a file's contents over the component objects.
Instead of limiting the system to simple striping
scheme where loss of a single component object results in
data loss, the map parameters support mirroring
and more complicated schemes that protect against loss
of a component object.
   </t>

   <t>
The stripe_unit is the number of bytes placed on one
component before advancing to the next one in the list
of components.  The number of bytes in a full stripe
is stripe_unit times the number of components.
In some raid schemes, a stripe includes redundant
information (i.e., parity) that lets the system
recover from loss or damage to a component object.
   </t>

   <t>
The group_width and group_depth parameters allow
a nested striping pattern.  If there is no nesting,
then group_width and group_depth MUST be zero.
Otherwise, the group_width defines the width
of a data stripe, and the group_depth defines
how many stripes are accessed before advancing
to the next group of components in the
list of component objects for the file.
The size of the components array MUST be a multiple
of group_width.
   </t>

   <t>
The mirror_cnt is used to replicate a file by replicating its component objects.
If there is no mirroring, then mirror_cnt MUST be 0.
If mirror_cnt is greater than zero, then the size of the component
array MUST be a multiple of (mirror_cnt+1).
   </t>

   <t>
See <xref target="Data Mapping Schemes" /> for more details.
   </t>
  </section>

 </section>

 <section anchor="Data Mapping Schemes" title="Data Mapping Schemes">
  <t>
This section describes the different data mapping schemes
in detail.
  </t>

  <section anchor="Simple Striping" title="Simple Striping">

   <t>
The object layout always uses a "dense"
layout as described in the pNFS document.
This means that the second stripe unit of the
file starts at offset 0 of the second component,
rather than at offset stripe_unit bytes.
After a full stripe has been written, the
next stripe unit is appended to the first component
object in the list without any holes in the component objects.
The mapping from the logical
offset within a file (L) to do the component object C and
object-specific offset O is defined by the following equations:
   </t>
   <figure>
    <artwork>
L = logical offset into the file
W = total number of components
S = W * stripe_unit
N = L / S
C = (L-(N*S)) / stripe_unit
O = (N*stripe_unit)+(L%stripe_unit)
    </artwork>
   </figure>
   <t>
In these equations, S is the number of bytes in a full stripe,
and N is the stripe number.  C is an index into the array of components,
so it selects a particular object storage device.
Both N and C count from zero.
O is the offset within the object that corresponds to the file offset.
Note that this computation does not accommodate the same
object appearing in the component array multiple times.
   </t>
   <t>
For example, consider an object striped over four devices, &lt;D0 D1 D2 D3&gt;.
The stripe_unit is 4096 bytes. The stripe width S is thus 4 * 4096 = 16384.
   </t>
   <figure>
    <artwork>
Offset 0:
  N = 0 / 16384 = 0
  C = 0-0/4096 = 0 (D0)
  O = 0*4096 + (0%4096) = 0

Offset 4096:
  N = 4096 / 16384 = 0
  C = (4096-(0*16384)) / 4096 = 1 (D1)
  O = (0*4096)+(4096%4096) = 0

Offset 9000:
  N = 9000 / 16384 = 0
  C = (9000-(0*16384)) / 4096 = 2 (D2)
  O = (0*4096)+(9000%4096) = 808

Offset 132000:
  N = 132000 / 16384 = 8
  C = (132000-(8*16384)) / 4096 = 0
  O = (8*4096) + (132000%4096) = 33696
    </artwork>
   </figure>
  </section>

  <section anchor="Nested Striping" title="Nested Striping">
   <t>
The group_width and group_depth parameters allow
a nested striping pattern.  If there is no nesting,
then group_width and group_depth MUST be zero.
Otherwise, the group_width defines the width
of a data stripe, and the group_depth defines
how many stripes are written before advancing
to the next group of components in the
list of component objects for the file.
The size of the components array MUST be a multiple
of group_width.
The math used to map from a file offset to
a component object and offset within that
object is shown below.
The computations
map from the logical offset L to the component index C and offset relative O
within that component object.
   </t>
   <figure>
    <artwork>
L = logical offset into the file
W = total number of components
S = stripe_unit * group_depth * W
T = stripe_unit * group_depth * group_width
U = stripe_unit * group_width
M = L / S
G = (L - (M * S)) / T
H = (L - (M * S)) % T
N = H / U
C = (H - (N * U)) / stripe_unit + G * group_width
O = L % stripe_unit + N * stripe_unit + M * group_depth * stripe_unit
    </artwork>
   </figure>
   <t>
In these equations, S is the number of bytes striped across all
component objects before the pattern repeats.  T is the number of bytes
striped within a group of component objects before advancing to the next group.
U is the number of bytes in a stripe within a group.
M is the "major" (i.e., across all components) stripe number,
and N is the "minor" (i.e., across the group) stripe number.
G counts the groups from the beginning of the major stripe,
and H is the byte offset within the group.
   </t>
   <t>
For example, consider an object striped over 100 devices with
a group_width of 10, a group_depth of 50, and a stripe_unit of 1 MB.
In this scheme, 500 MB are written to the first 10 components,
and 5000 MB is written before the pattern wraps back around to the
first component in the array.
   </t>
   <figure>
    <artwork>
Offset 0:
  W = 100
  S = 1 MB * 50 * 100 = 5000 MB
  T = 1 MB * 50 * 10 = 500 MB
  U = 1 MB * 10 = 10 MB
  M = 0 / 5000 MB = 0
  G = (0 - (0 * 5000 MB)) / 500 MB = 0
  H = (0 - (0 * 5000 MB)) % 500 MB = 0
  N = 0 / 10 MB = 0
  C = (0 - (0 * 10 MB)) / 1 MB + 0 * 10 = 0
  O = 0 % 1 MB + 0 * 1 MB + 0 * 50 * 1 MB = 0

Offset 27 MB:
  M = 27 MB / 5000 MB = 0
  G = (27 MB - (0 * 5000 MB)) / 500 MB = 0
  H = (27 MB - (0 * 5000 MB)) % 500 MB = 27 MB
  N = 27 MB / 10 MB = 2
  C = (27 MB - (2 * 10 MB)) / 1 MB + 0 * 10 = 7
  O = 27 MB % 1 MB + 2 * 1 MB + 0 * 50 * 1 MB = 2 MB

Offset 7232 MB:
  M = 7232 MB / 5000 MB = 1
  G = (7232 MB - (1 * 5000 MB)) / 500 MB = 4
  H = (7232 MB - (1 * 5000 MB)) % 500 MB = 232 MB
  N = 232 MB / 10 MB = 23
  C = (232 MB - (23 * 10 MB)) / 1 MB + 4 * 10 = 42
  O = 7232 MB % 1 MB + 23 * 1 MB + 1 * 50 * 1 MB = 73 MB
    </artwork>
   </figure>
  </section>

  <section anchor="Mirroring" title="Mirroring">
   <t>
The mirror_cnt is used to replicate a file by replicating its component objects.
If there is no mirroring, then mirror_cnt MUST be 0.
If mirror_cnt is greater than zero, then the size of the component
array MUST be a multiple of (mirror_cnt+1).
Thus, for a classic mirror on two objects, mirror_cnt is one.
If group_width is also non-zero, then the size MUST be a
multiple of group_width * (mirror_cnt+1).
Replicas are adjacent in the components array,
and the value C produced by the above equations is not
a direct index into the components array.  Instead,
the following equations determine the replica component index RCi,
where i ranges from 0 to mirror_cnt.
   </t>
   <figure>
    <artwork>
C = component index for striping or two-level striping
i ranges from 0 to mirror_cnt, inclusive
RCi = C * (mirror_cnt+1) + i
    </artwork>
   </figure>
  </section>
 </section>

 <section anchor="RAID Algorithms" title="RAID Algorithms">
  <t>
pnfs_osd_raid_algorithm4 determines the algorithm and placement of
redundant data.
This section defines the different RAID algorithms.
  </t>

  <section anchor="PNFS_OSD_RAID_0" title="PNFS_OSD_RAID_0">
   <t>
PNFS_OSD_RAID_0 means there is no
parity data, so all bytes in the component objects are
data bytes located by the above equations for C and O.
If a component object is unavailable, the pNFS client can
choose to return NULLs for the missing data, or it can
retry the READ against the pNFS server, or it can return
an EIO error.
   </t>
  </section>

  <section anchor="PNFS_OSD_RAID_4" title="PNFS_OSD_RAID_4">
   <t>
PNFS_OSD_RAID_4 means that the last component object,
or the last in each group if group_width is > zero,
contains parity information computed over the rest of
the stripe with an XOR operation.
If a component object is unavailable, the client can
read the rest of the stripe units in the damaged stripe
and recompute the missing stripe unit by XORing the other
stripe units in the stripe.  Or the client can replay
the READ against the pNFS server which will presumably
perform the reconstructed read on the client's behalf.
   </t>
   <t>
When parity is present in the file,
then there is an additional computation to map from the file offset L
to the offset that accounts for embedded parity, L'.
First compute L', and then use L' in the above equations for C and O.
   </t>
   <figure>
    <artwork>
L = file offset, not accounting for parity
P = number of parity devices in each stripe
W = group_width, if not zero, else size of component array
N = L / (W-P * stripe_unit)
L' = N * (W * stripe_unit) +
     (L % (W-P * stripe_unit))
    </artwork>
   </figure>
  </section>

  <section anchor="PNFS_OSD_RAID_5" title="PNFS_OSD_RAID_5">
   <t>
PNFS_OSD_RAID_5 means that the position of the parity data
is rotated on each stripe.  In the first stripe, the last
component holds the parity.  In the second stripe, the
next-to-last component holds the parity, and so on.
In this scheme, all stripe units are rotated so that I/O
is evenly spread across objects as the file is read
sequentially.  The rotated parity layout is illustrated here,
with numbers indicating the stripe unit.
   </t>
   <figure>
    <artwork>
0 1 2 P
4 5 P 3
8 P 6 7
P 9 a b
    </artwork>
   </figure>
   <t>
To compute the component object C, first compute the
offset that accounts for parity L' and use that to
compute C.  Then rotate C to get C'.
Finally, increase C' by one if the parity
information comes at or before C' within that stripe.
The following equations illustrate this by computing I,
which is the index of the component that contains
parity for a given stripe.
   </t>
   <figure>
    <artwork><![CDATA[
L = file offset, not accounting for parity
W = group_width, if not zero, else size of component array
N = L / (W-1 * stripe_unit)
(Compute L' as describe above)
(Compute C based on L' as described above)
C' = (C - (N%W)) % W
I = W - (N%W) - 1
if (C' <= I) {
  C'++
}
]]></artwork>
   </figure>
  </section>

  <section anchor="PNFS_OSD_RAID_PQ" title="PNFS_OSD_RAID_PQ">
   <t>
PNFS_OSD_RAID_PQ is a double-parity scheme that uses
the Reed-Solomon P+Q encoding scheme.  In this layout,
the last two component objects hold the P and Q data,
respectively.  P is parity computed with XOR, and
Q is a more complex equation that is not described here.
The equations given above for embedded parity can be
used to map a file offset to the correct component
object by setting the number of parity components to 2
instead of 1 for RAID4 or RAID5.
Clients may simply choose to read data through the
metadata server if two components are missing or
damaged.
   </t>
   <t>
Issue: This scheme also has a RAID_4 like layout where
the ECC blocks are stored on the same components on every stripe
and a rotated, RAID-5 like layout where the stripe units are rotated.
Should we make the following properties orthogonal: RAID_4 or RAID_5 (i.e.,
non-rotated or rotated), and then have the number of parity components
and the associated algorithm be the orthogonal parameter?
   </t>
  </section>

  <section title="RAID Usage and implementation notes">
   <t>
RAID layouts with redundant data in their stripes
require additional serialization of updates to
ensure correct operation. Otherwise, if two clients simultaneously
write to the same logical range of an object, the result could include
different data in the same ranges of mirrored tuples, or corrupt parity
information.  It is the
responsibility of the metadata server to enforce serialization
requirements such as this. For example, the metadata server may do
so by not granting overlapping write layouts within mirrored objects.
   </t>
  </section>
 </section>
</section>

<section title="Object-Based Layout Update">
 <t>
layoutupdate4 is used in the LAYOUTCOMMIT operation
to convey updates to the layout and additional information to
the metadata server.  It is defined in the
<xref target="NFSv4.1">NFSv4.1 draft</xref> as follows:
 </t>

 <figure>
  <artwork>
struct layoutupdate4 {
    layouttype4             lou_type;
    opaque                  lou_body&lt;&gt;;
};
  </artwork>
 </figure>

 <t>
The layoutupdate4 type is an opaque value at the generic pNFS client level.
If the lou_type layout type is LAYOUT4_OSD2_OBJECTS, then
the lou_body opaque value is defined by the pnfs_osd_layoutupdate4 type.
 </t>

 <section anchor="pnfs_osd_layoutupdate4" title="pnfs_osd_layoutupdate4">
  <figure>
   <artwork>
struct pnfs_osd_layoutupdate4 {
    pnfs_osd_deltaspaceused4    delta_space_used;
    pnfs_osd_ioerr4             ioerr&lt;&gt;;
};
   </artwork>
  </figure>

  <t>
Object-Based pNFS clients are not allowed to modify the layout.
"delta_space_used" is used to convey capacity usage
information back to the metadata server and,
in case OSD I/O operations failed,
"ioerr" is used to report these errors to the metadata server.
  </t>

  <section anchor="pnfs_osd_deltaspaceused4" title="pnfs_osd_deltaspaceused4">
   <figure>
    <artwork>
union pnfs_osd_deltaspaceused4 switch (bool valid) {
    case TRUE:
        int64_t     delta;  /* Bytes consumed by write activity */
    case FALSE:
        void;
};
    </artwork>
   </figure>
   <t>
pnfs_osd_deltaspaceused4 is used to convey space utilization information
at the time of LAYOUTCOMMIT.  For the file system to properly maintain
capacity used information, it needs to track how much capacity was
consumed by WRITE operations performed by the client.  In this protocol,
the OSD returns the capacity consumed by a write, which can be
different than the number of bytes written
because of internal overhead like block-based allocation and
indirect blocks, and the client reflects this back to the pNFS server
so it can accurately track quota.  The pNFS server can choose to
trust this information coming from the clients and therefore
avoid querying the OSDs at the time of LAYOUTCOMMIT.
If the client is unable to obtain this information from the OSD,
it simply returns invalid delta_space_used.
   </t>
  </section>

  <section anchor="pnfs_osd_errno4" title="pnfs_osd_errno4">
   <figure>
    <artwork>
enum pnfs_osd_errno4 {
    PNFS_OSD_ERR_EIO            = 1,
    PNFS_OSD_ERR_NOT_FOUND      = 2,
    PNFS_OSD_ERR_NO_SPACE       = 3,
    PNFS_OSD_ERR_BAD_CRED       = 4,
    PNFS_OSD_ERR_NO_ACCESS      = 5,
    PNFS_OSD_ERR_UNREACHABLE    = 6,
    PNFS_OSD_ERR_RESOURCE       = 7
};
    </artwork>
   </figure>

   <t>
pnfs_osd_errno4 is used to represent error types when read/write errors
are reported to the metadata server.
The error codes serve as hints to the metadata server that may help it
in diagnosing the exact reason for the error and in repairing it.

   <list style="symbols">
    <t>
PNFS_OSD_ERR_EIO indicates the operation failed because the
Object Storage Device experienced a failure trying to access the
object.  The most common source of these errors is media errors,
but other internal errors might cause this. In this case,
the metadata server should go examine the broken object 
more closely, hence it should be used as the default error code.
    </t>
    <t>
PNFS_OSD_ERR_NOT_FOUND indicates the object ID specifies an object that
does not exist on the Object Storage Device.
    </t>
    <t>
PNFS_OSD_ERR_NO_SPACE indicates the operation failed because the
Object Storage Device ran out of free capacity during the operation.
    </t>
    <t>
PNFS_OSD_ERR_BAD_CRED indicates the security parameters are not valid.
The primary cause of this is that the capability has expired,
or the access policy tag (a.k.a, capability version number) has
been changed to revoke capabilities.  The client will need to
return the layout and get a new one with fresh capabilities.
    </t>
    <t>
PNFS_OSD_ERR_NO_ACCESS indicates the capability does not allow
the requested operation.  This should not occur in normal operation
because the metadata server should give out correct capabilities,
or none at all.
    </t>
    <t>
PNFS_OSD_ERR_UNREACHABLE indicates the client did not complete
the I/O operation at the Object Storage Device
due to a communication failure.  Whether the I/O operation
was executed by the OSD or not is undetermined.
    </t>
    <t>
PNFS_OSD_ERR_RESOURCE indicates the client did not issue
the I/O operation due to a local problem on the initiator (i.e. client)
side, e.g., when running out of memory.  The client
MUST guarantee that the OSD command was never dispatched to the OSD.
    </t>
   </list>
   </t>
  </section>

  <section anchor="pnfs_osd_ioerr4" title="pnfs_osd_ioerr4">
   <figure>
    <artwork>
struct pnfs_osd_ioerr4 {
    pnfs_osd_objid4     component;
    length4             comp_offset;
    length4             comp_length;
    bool                iswrite;
    pnfs_osd_errno4     errno;
};
    </artwork>
   </figure>
   <t>
The pnfs_osd_ioerr4 structure is used to return error indications
for objects that generated errors during data transfers.
These are hints to the
metadata server that there are problems with that object.
For each error, "component", "comp_offset", and "comp_length" represent the object
and byte range within the component object in which the error occurred.
"iswrite" is set to "true" if the failed OSD operation was data modifying,
and "errno" represents the type of error.
   </t>
  </section>

 </section>
</section>

<section title="Object-Based Creation Layout Hint">
 <t>
The layouthint4 type is defined in the 
<xref target="NFSv4.1">NFSv4.1 draft</xref> as follows:
 </t>

 <figure>
  <artwork>
struct layouthint4 {
    layouttype4           loh_type;
    opaque                loh_body&lt;&gt;;
};
  </artwork>
 </figure>

 <t>
The layouthint4 structure is used by the client to pass in a
hint about the type of layout it would like created for a particular
file.
If the loh_type layout type is LAYOUT4_OSD2_OBJECTS, then
the loh_body opaque value is defined by the pnfs_osd_layouthint4 type.
 </t>

 <section anchor="pnfs_osd_layouthint4" title="pnfs_osd_layouthint4">

  <figure>
   <artwork>
union num_comps_hint4 switch (bool valid) {
    case TRUE:
        uint32_t            num_comps;
    case FALSE:
        void;
};

union stripe_unit_hint4 switch (bool valid) {
    case TRUE:
        length4             stripe_unit;
    case FALSE:
        void;
};

union group_width_hint4 switch (bool valid) {
    case TRUE:
        uint32_t            group_width;
    case FALSE:
        void;
};

union group_depth_hint4 switch (bool valid) {
    case TRUE:
        uint32_t            group_depth;
    case FALSE:
        void;
};

union mirror_cnt_hint4 switch (bool valid) {
    case TRUE:
        uint32_t            mirror_cnt;
    case FALSE:
        void;
};

union raid_algorithm_hint4 switch (bool valid) {
    case TRUE:
        pnfs_osd_raid_algorithm4    raid_algorithm;
    case FALSE:
        void;
};

struct pnfs_osd_layouthint4 {
    num_comps_hint4         num_comps_hint;
    stripe_unit_hint4       stripe_unit_hint;
    group_width_hint4       group_width_hint;
    group_depth_hint4       group_depth_hint;
    mirror_cnt_hint4        mirror_cnt_hint;
    raid_algorithm_hint4    raid_algorithm_hint;
};
   </artwork>
  </figure>

  <t>
This type conveys hints for the desired data map.
All parameters are optional so the client can give values for only
the parameters it cares about, e.g. it can provide a hint for the desired
number of mirrored components, regardless of the the raid algorithm selected
for the file.  The server should make an attempt to honor the hints 
but it can ignore any or all of them at its own discretion and
without failing the respective create operation.
  </t>

  <t>
The num_comps hint can be used to limit the total number of component
objects comprising the file.  All other hints correspond directly to the
different fields of pnfs_osd_data_map4.
  </t>

 </section>
</section>

<section title="Layout Segments">
 <t>
The pnfs layout operations operate on logical byte ranges.
There is no requirement in the protocol for any relationship between
byte ranges used in LAYOUTGET to acquire layouts and byte ranges used
in CB_LAYOUTRECALL, LAYOUTCOMMIT, or LAYOUTRETURN.
However, using OSD capabilities poses limitations on these operations
since the capabilities associated with layout segments cannot be merged
or split.
The following guidelines should be followed for proper operation of
object-based layouts.
 </t>

 <section title="CB_LAYOUTRECALL and LAYOUTRETURN">

  <t>
In general, the object-based layout driver should keep track of each
layout segment it got, keeping record of the segment's iomode, offset,
and length.
The server should allow the client to get multiple overlapping layout
segments but is free to recall the layout to prevent overlap.
  </t>

  <t>
In response to CB_LAYOUTRECALL, the client should return all layout
segments matching the given iomode and overlapping with the recalled range.
When returning the layouts for this byte range with LAYOUTRETURN the client
MUST NOT return a sub-range of a layout segment it has;
each LAYOUTRETURN sent MUST completely cover at least one outstanding layout
segment.
  </t>

  <t>
The server, in turn, should release any segment that exactly matches the
clientid, iomode, and byte range given in LAYOUTRETURN.
If no exact match is found then the server should release all layout segments
matching the clientid and iomode and that are fully contained in the
returned byte range.
If none are found and the byte range is a subset of an outstanding layout
segment with for the same clientid and iomode, then the client can be
considered malfunctioning and the server SHOULD recall all layouts from
this client to reset its state.  If this behavior repeats the server
SHOULD deny all LAYOUTGETs from this client.
  </t>
 </section>

 <section title="LAYOUTCOMMIT">
  <t>
LAYOUTCOMMIT is only used by object-based pNFS to convey modified attributes
hints and/or to report I/O errors to the MDS.
Therefore, the offset and length in LAYOUTCOMMIT4args are reserved for future
use and should be set to 0.
However, component byte ranges in the optional pnfs_osd_ioerr4 structure are
used for recovering the object and MUST be set by the client to cover all
failed I/O operations to the component.
  </t>
 </section>
</section>

<section title="Recalling Layouts">
 <t>
The object-based metadata server should recall outstanding layouts
in the following cases:

 <list style='symbols'>
  <t>
  When the file's security policy changes, i.e. ACLs or permission mode bits
are set.
  </t>
  <t>
  When the file's aggregation map changes, rendering outstanding layouts invalid.
  </t>
  <t>
  When there are sharing conflicts. For example, the server will issue
stripe aligned layout segments for RAID-5 objects.  To prevent corruption
of the file's parity, Multiple clients must not hold valid write layouts
for the same stripes.
An outstanding RW layout should be recalled when a conflicting LAYOUTGET
is received from a different client for LAYOUTIOMODE4_RW and for a byte-range
overlapping with the outstanding layout segment.
  </t>
 </list>
 </t>

 <section title="CB_RECALL_ANY" anchor="CB_RECALL_ANY">
  <t>
The metadata server can use the CB_RECALL_ANY callback operation to notify
the client to return some or all of its layouts.
The <xref target="NFSv4.1">NFSv4.1 draft</xref> defines
the following types:
  </t>

  <figure>
   <artwork>
const RCA4_TYPE_MASK_OBJ_LAYOUT_MIN     = 8;
const RCA4_TYPE_MASK_OBJ_LAYOUT_MAX     = 11;

struct  CB_RECALL_ANY4args      {
    uint32_t        craa_objects_to_keep;
    bitmap4         craa_type_mask;
};
   </artwork>
  </figure>

  <t>
Typically, CB_RECALL_ANY will be used to recall client state when the server
needs to reclaim resources. The craa_type_mask bitmap specifies the type of
resources that are recalled and the craa_objects_to_keep value specifies
how many of the recalled objects the client is allowed to keep.

The object-based layout type mask flags are defined as follows.
They represent the iomode of the recalled layouts.
In response, the client SHOULD return layouts of the recalled iomode
that it needs the least,
keeping at most craa_objects_to_keep object-based layouts.
  </t>
  <figure>
   <artwork>
const PNFS_OSD_RCA4_TYPE_MASK_READ  = RCA4_TYPE_MASK_OBJ_LAYOUT_MIN;
const PNFS_OSD_RCA4_TYPE_MASK_RW    = RCA4_TYPE_MASK_OBJ_LAYOUT_MIN+1;
const PNFS_OSD_RCA4_TYPE_MASK_ANY   = RCA4_TYPE_MASK_OBJ_LAYOUT_MIN+2;
   </artwork>
  </figure>

  <t>
The PNFS_OSD_RCA4_TYPE_MASK_READ flag notifies the client to return layouts
of iomode LAYOUTIOMODE4_READ.
Similarly, the PNFS_OSD_RCA4_TYPE_MASK_RW flag notifies the client to return layouts
of iomode LAYOUTIOMODE4_RW.
The PNFS_OSD_RCA4_TYPE_MASK_ANY flag notifies the client to return layouts
of either iomode.
  </t>
 </section>
</section>

<section title="Client Fencing" anchor="Client Fencing">
 <t>
In cases where clients are uncommunicative and their lease has expired
or when clients fail to return recalled layouts in a timely manner the
server MAY revoke client layouts and/or device address mappings and reassign
these resources to other clients.
To avoid data corruption, the metadata server MUST fence off the revoked
clients from the respective objects as described in <xref target="Revoking Capabilities" />.
 </t>
</section>

<section title="Security Considerations" anchor="Security Considerations">
 <t>
  The pNFS extension partitions the NFSv4 file system protocol into
  two parts, the control path and the data path (storage protocol).
  The control path contains all the new operations described by this
  extension; all existing NFSv4 security mechanisms and features apply
  to the control path.  The combination of components in a pNFS system
  is required to preserve the
  security properties of NFSv4 with respect to an entity accessing
  data via a client, including security countermeasures to defend
  against threats that NFSv4 provides defenses for in environments
  where these threats are considered significant.
 </t>
 <t>
  The metadata server enforces the file access-control policy at LAYOUTGET time.
  The client should use suitable authorization credentials for getting the
  layout for the requested iomode (READ or RW) and the server verifies the
  permissions and ACL for these credentials, possibly returning NFS4ERR_ACCESS
  if the client is not allowed the requested iomode.  If the LAYOUTGET
  operation succeeds the client receives, as part of the layout, a set of
  object capabilities allowing it I/O access to the specified objects
  corresponding to the requested iomode.  When the client acts on I/O operations
  on behalf of its local users it MUST authenticate and authorize the user by
  issuing respective OPEN and ACCESS calls to the metadata server, similarly
  to having NFSv4 data delegations.  If access is allowed the client uses the
  corresponding (READ or RW) capabilities to perform the I/O operations at the
  object-storage devices.  When the metadata server receives a request to change
  file's permissions or ACL it SHOULD recall all layouts for that file
  and it MUST change the capability version attribute on all objects comprising
  the file to implicitly invalidate any outstanding capabilities before
  committing to the new permissions and ACL.  Doing this will ensure that
  clients re-authorize their layouts according to the modified permissions and
  ACL by requesting new layouts.  Recalling the layouts in this case is courtesy
  of the server intended to prevent clients from getting an error on I/Os done
  after the capability version changed.
 </t>
 <t>
   The object storage protocol MUST implement the security aspects
   described in version 1 of the T10
   <xref target="osd standard">OSD protocol definition</xref>.
  The standard defines four security methods: NOSEC, CAPKEY, CMDRSP,
  and ALLDATA.  To provide minimum level of security allowing verification
  and enforcement of the server access control policy using the layout
  security credentials, the NOSEC security method MUST NOT be used for
  I/O operation. It MAY only be used to get the System ID attribute when the
  metadata server provided only the OSD name with the device address.

   The remainder of this section gives an overview of the security mechanism
   described in that standard.  The goal is to give the reader a basic
   understanding of the object security model.  Any discrepancies
   between this text and the actual standard are obviously to be
   resolved in favor of the OSD standard.
 </t>

 <section anchor="OSD Security Data Types" title="OSD Security Data Types">
  <t>
There are three main data types associated with object security:
a capability, a credential, and security parameters.
The capability is a set of fields that specifies an object
and what operations can be performed on it.
A credential is a signed capability.  Only a security manager
that knows the secret device keys can correctly sign a capability
to form a valid credential.
In pNFS, the file server acts as the security manager and
returns signed capabilities (i.e., credentials) to the pNFS client.
The security parameters are values computed by the issuer of OSD
commands (i.e., the client) that prove they hold valid credentials.
The client uses the credential as a signing key to sign the
requests it makes to OSD, and puts the resulting signatures
into the security_parameters field of the OSD command.
The object storage device uses the secret keys it shares with
the security manager to validate the signature values in
the security parameters.
  </t>
  <t>
The security types are opaque to the generic layers of the
pNFS client.
The credential contents are defined as opaque within the pnfs_osd_object_cred4 
type.
Instead of repeating the definitions here,
the reader is referred to section 4.9.2.2 of the OSD standard.
  </t>
 </section>
 <section title="The OSD Security Protocol">
  <t>
  The object storage protocol relies on a cryptographically secure
  capability to control accesses at the object storage devices.
  Capabilities are generated by the metadata server, returned to the
  client, and used by the client as described below to authenticate
  their requests to the Object Storage Device (OSD).  Capabilities
  therefore achieve the required access and open mode checking.  They
  allow the file server to define and check a policy (e.g., open mode)
  and the OSD to enforce that policy without knowing
  the details (e.g., user IDs and ACLs).
  </t>
  <t>
  Since capabilities are tied to layouts, and since they are used to
  enforce access control, when the file ACL or mode changes the outstanding
  capabilities MUST be revoked to enforce the new access permissions.
  The server SHOULD recall layouts to allow clients to gracefully
  return their capabilities before the access permissions change.
  </t>
  <t>
  Each capability is specific to a particular object, an operation
  on that object, a byte range w/in the object (in OSDv2), and has an explicit
  expiration time.  The capabilities are signed with a secret key
  that is shared by the object storage devices (OSD) and the metadata
  managers.  Clients do not have device keys so they are unable to
  forge the signatures in the security parameters.
  The combination of a capability, the OSD system id, and a signature is called
  a "credential" in the OSD specification.
  </t>
  <t>
  The details of the security and privacy model for Object Storage
  are defined in the T10 OSD standard.
  The following sketch of the algorithm should help the
  reader understand the basic model.
  </t>
  <t>
  LAYOUTGET returns a CapKey and a Cap which, together with the OSD SystemID,
  are also called a credential.
  It is a capability and a signature over that capability and the SystemID.
  The OSD Standard refers to the CapKey as the "Credential integrity
  check value" and to the ReqMAC as the "Request integrity check value".
   <figure>
    <artwork><![CDATA[
CapKey = MAC<SecretKey>(Cap, SystemID)
Credential = {Cap, SystemID, CapKey}
]]></artwork>
   </figure>
  The client uses CapKey to sign all the requests it issues
  for that object using the respective Cap.  In other words,
  the Cap appears in the request to the storage device, and
  that request is signed with the CapKey as follows:
   <figure>
    <artwork><![CDATA[
ReqMAC = MAC<CapKey>(Req, ReqNonce)
Request = {Cap, Req, ReqNonce, ReqMAC}
]]></artwork>
   </figure>
  The following is sent to the OSD: {Cap, Req, ReqNonce, ReqMAC}.
  The OSD uses the SecretKey it shares with the metadata server
  to compare the ReqMAC the client sent with a locally computed value:
   <figure>
    <artwork><![CDATA[
LocalCapKey = MAC<SecretKey>(Cap, SystemID)
LocalReqMAC = MAC<LocalCapKey>(Req, ReqNonce)
]]></artwork>
   </figure>
  and if they match the
  OSD assumes that the capabilities came from an authentic
  metadata server and allows access to the object, as allowed
  by the Cap.
  </t>
 </section>

 <section title="Protocol Privacy Requirements">
  <t>
  Note that if the server LAYOUTGET reply,
  holding CapKey and Cap, is snooped by another client, it can
  be used to generate valid OSD requests (within the Cap
  access restrictions).
  </t>
  <t>
  To provide the required privacy requirements for the capability key
  returned by LAYOUTGET, the GSS-API can be used, e.g. by using the RPCSEC_GSS
  privacy method to send the LAYOUTGET operation or by
  using the SSV key to encrypt the capability_key using the GSS_Wrap() function.
  Two general ways to
  provide privacy in the absence of GSS-API that are independent
  of NFSv4 are either an isolated network such as a VLAN or a
  secure channel provided by <xref target='RFC4301'>IPsec</xref>.
  </t>
 </section>

 <section title="Revoking Capabilities" anchor="Revoking Capabilities">
  <t>
At any time, the metadata server may invalidate all outstanding
capabilities on an object by changing its POLICY ACCESS TAG attribute.
The value of the POLICY ACCESS TAG is part of a capability, and it
must match the state of the object attribute.  If they do not match,
the OSD rejects accesses to the object with the sense key set to
ILLEGAL REQUEST and an additional sense code set to INVALID FIELD IN CDB.
When a client attempts to use a capability and is rejected
this way, it should issue a LAYOUTCOMMIT
for the object and specify PNFS_OSD_BAD_CRED in the ioerr
parameter. The client may elect to issue a compound
LAYOUTRETURN/LAYOUTGET (or LAYOUTCOMMIT/LAYOUTRETURN/LAYOUTGET)
to attempt to fetch a refreshed set of capabilities.
  </t>
  <t>
The metadata server may elect to change the access policy tag
on an object at any time, for any reason (with the understanding
that there is likely an associated performance penalty, especially
if there are outstanding layouts for this object). The metadata
server MUST revoke outstanding capabilities when any one of
the following occurs:
   <list style='symbols'>
    <t>the permissions on the object change,</t>
    <t>a conflicting mandatory byte-range lock is granted, or</t>
    <t>a layout is revoked and reassigned to another client</t>
   </list>
  </t>
  <t>
A pNFS client will typically hold one layout for each byte range for either
READ or READ/WRITE. The client's credentials are checked by the metadata server
at LAYOUTGET time and it is the client's responsibility to enforce access
control among multiple users accessing the same file. It is neither required
nor expected that the pNFS client will obtain a separate layout for each user
accessing a shared object. The client SHOULD use OPEN and ACCESS calls to check user
permissions when performing I/O so that the server's access control policies
are correctly enforced. The result of the ACCESS operation may be cached
while the client holds a valid layout as the server is expected to recall
layouts when the file's access permissions or ACL change.
  </t>

 </section>
</section>

<section anchor="IANA Considerations" title="IANA Considerations">
 <t>
As described in the <xref target="NFSv4.1">NFSv4.1 draft</xref>,
new layout type numbers will be requested from IANA.
This document defines the protocol associated with the existing
layout type number, LAYOUT4_OSD2_OBJECTS, and it requires no further
actions for IANA.
 </t>
</section>

</middle>

<back>

<references title="Normative References">
  <reference anchor='osd standard' 
             target='http://www.t10.org/ftp/t10/drafts/osd/osd-r10.pdf'>
    <front>
      <title>SCSI Object-Based Storage Device Commands</title>
      <author initials="R." surname="Weber" fullname="Ralph Weber">
        <organization abbrev="SNIA">SNIA T10/1355-D</organization>
      </author>
      <date month="July" year="2004"/>
    </front>
  </reference>

    <reference anchor='RFC2119'>
      <front>
      <title abbrev='RFC Key Words'>Key words for use in RFCs to Indicate Requirement Levels</title>
      <author initials='S.' surname='Bradner' fullname='Scott Bradner'>
      <organization>Harvard University</organization>

      <address>
      <postal>
      <street>1350 Mass. Ave.</street>
      <street>Cambridge</street>
      <street>MA 02138</street></postal>
      <phone>- +1 617 495 3864</phone>
      <email>[email-masked]</email></address></author>
      <date year='1997' month='March' />
      </front>
      <seriesInfo name="RFC" value="2119"/>
      <format type="TXT" octets="4723" 
       target="http://www.ietf.org/rfc/rfc2119.txt"/>
    </reference>
 
    <reference anchor='RFC4506'>
      <front>
      <title abbrev='XDR'>XDR: External Data Representation Standard</title>
      <author initials='M.' surname='Eisler' fullname='Mike Eisler'>
      <organization>Network Appliance, Inc.</organization>
      </author>
      <date month='May' year='2006'/>
      </front>
      <seriesInfo name='STD' value='67' />
      <seriesInfo name="RFC" value="4506"/>
      <format type="TXT" octets="55477" 
       target="http://www.ietf.org/rfc/rfc4506.txt"/>
    </reference>
 
    <reference anchor="RFC3530">
	  <front>
      <title>Network File System (NFS) version 4 Protocol</title>
	  <author initials="S." surname="Shepler" fullname="S. Shepler">
        <organization>Sun Microsystems, Inc.</organization>
      </author>
	  <author initials="B." surname="Callaghan" fullname="B. Callaghan">
        <organization>Sun Microsystems, Inc.</organization>
      </author>
	  <author initials="D." surname="Robinson" fullname="D. Robinson">
        <organization>Sun Microsystems, Inc.</organization>
      </author>
	  <author initials="R." surname="Thurlow" fullname="R. Thurlow">
        <organization>Sun Microsystems, Inc.</organization>
      </author>
	  <author initials="C." surname="Beame" fullname="C. Beame">
        <organization>Hummingbird, Ltd.</organization>
      </author>
	  <author initials="M." surname="Eisler" fullname="M. Eisler">
        <organization>Network Appliance, Inc.</organization>
      </author>
	  <author initials="D." surname="Noveck" fullname="D. Noveck">
        <organization>Network Appliance, Inc.</organization>
      </author>
      <date year="2003" month="April"/>
      </front>
      <seriesInfo name="RFC" value="3530"/>
      <format type="TXT" octets="600986" 
       target="http://www.ietf.org/rfc/rfc3530.txt"/>
    </reference>

    <reference anchor='RFC2743'>

      <front>
	<title abbrev='GSS-API'>Generic Security Service Application
	Program Interface Version 2, Update 1</title>
	<author initials='J.' surname='Linn' fullname='John Linn'>
	  <organization>RSA Laboratories</organization>
	  <address>
	    <postal>
	      <street>20 Crosby Drive</street>
	      <city>Bedford</city>
	      <region>MA</region>
	      <code>01730</code>
	    <country>US</country></postal>
	    <phone>+1 781 687 7817</phone>
	<email>[email-masked]</email></address></author>
	<date year='2000' month='January' />
	<abstract>
	  <t>The Generic Security Service Application Program
	  Interface (GSS-API), Version 2, as defined in [RFC-2078], provides
	  security services to callers in a generic fashion,
	  supportable with a range of underlying mechanisms and
	  technologies and hence allowing source-level portability of
	  applications to different environments. This specification
	  defines GSS-API services and primitives at a level
	  independent of underlying mechanism and programming language
	  environment, and is to be complemented by other, related
	  specifications:</t>
	  <t>documents defining specific parameter bindings for
	  particular language environments</t>
	  <t>documents defining token formats, protocols, and
	  procedures to be implemented in order to realize GSS-API
	  services atop particular security mechanisms</t> <t>This
	  memo obsoletes [RFC-2078], making specific, incremental changes in
	  response to implementation experience and liaison
	  requests. It is intended, therefore, that this memo or a
	  successor version thereto will become the basis for
	  subsequent progression of the GSS-API specification on the
	  standards track.</t></abstract></front>

      <seriesInfo name='RFC' value='2743' />
      <format type='TXT' octets='229418' target='ftp://ftp.isi.edu/in-notes/rfc2743.txt' />
    </reference>

</references>
<references title="Informative References">

  <reference anchor='NFSv4.1' 
             target='http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-minorversion1-13.txt'>
    <front>
      <title>NFSv4 Minor Version 1</title>
      <author initials="S." surname="Shepler" fullname="Spencer Shepler">
        <organization abbrev="Sun">Sun Microsystems, Inc.</organization>
      </author>
      <author initials="M." surname="Eisler" fullname="Mike Eisler">
        <organization abbrev="Netapp">Network Appliance, Inc.</organization>
      </author>
      <author initials="D." surname="Noveck" fullname="David Noveck">
        <organization abbrev="Netapp">Network Appliance, Inc.</organization>
      </author>
      <date month="March" year="2007"/>
    </front>
  </reference>

  <reference anchor='OSD2'
             target='http://www.t10.org/ftp/t10/drafts/osd2/osd2r02.pdf'>
    <front>
      <title>SCSI Object-Based Storage Device Commands -2 (OSD-2)</title>
      <author initials="R." surname="Weber" fullname="Ralph O. Weber">
        <organization abbrev="SNIA">SNIA T10/1729-D</organization>
      </author>
      <date month="January" year="2007"/>
    </front>
  </reference>

  <reference anchor='RFC4301'>
    <front>
      <title>Security Architecture for the Internet Protocol</title>
      <author initials='S.' surname='Kent' fullname='S. Kent'>
        <organization>BBN Technologies</organization>
      </author>
      <author initials='K.' surname='Seo' fullname='K. Seo'>
        <organization>BBN Technologies</organization>
      </author>
      <date year='2005' month='December'/>
    </front>
    <seriesInfo name='RFC' value='4301'/>
    <format type='TXT' octets='262123'
      target='http://www.ietf.org/rfc/rfc4301.txt'/>
  </reference>

</references>

<section title="Acknowledgments">
 <t>
    Todd Pisek was a co-editor of the initial drafts for this document.
 </t>
</section>
</back>
</rfc>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4