Subject: [nfsv4] eia_flags and eir_flags: Exchange id

From: Mahesh Siddheshwar <[email-masked]>
Date: Fri, 01 Feb 2008 13:30:39 -0800

During discussions regarding the server role bits --
(EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS
| EXCHID4_FLAG_USE_PNFS_DS) in eia_flags/eir_flags of EXCHANGE_ID,
we felt that there is a need for clarification of text regarding the 
bits set
in the request and server results. Below is the suggested addition to the
text in the specified sections (diffs attached).

I also propose that the client MUST state a preference of the
interested server role, ie. set at least one of the bits instead
of allowing zero. This will simplify and avoid confusion in the bits
the server should set in the results (and use any priority).
Specifically in the case of a server that has multiple concurrent
PNFS roles, each role served by a unique client ID.   Comments?
(this proposal is not in the suggested text changes below)

The first suggested change is in the EXCHANGE_ID section,
18.35.3 --  added:

26218,26226c26240,26249
<    convey roles the client ID is to be used for in a pNFS environment
<    Note that the same client owner/server owner pair can have multiple
<    roles.  Multiple roles can be associated with the same client ID or
<    with different client IDs.  Thus, if a client sends EXCHANGE_ID from
<    the same client owner to the same server owner multiple times, but
<    specifies different pNFS roles each time, the server might return
<    different client IDs.  Given that different pNFS roles might have
<    different client IDs, the client may ask for different properties for
<    each role/client ID.
---
 >    convey roles the client ID is to be used for in a pNFS environment.
 >    The server MUST set one of the acceptable combination of these bits
 >    in eir_flags, as specified in Section 13.1. Note that the same
        (^^^^^ added the above line)
 >    client owner/server owner pair can have multiple roles. Multiple roles
 >    can be associated with the same client ID or with different client 
IDs.
 >    Thus, if a client sends EXCHANGE_ID from the same client owner to the
 >    same server owner multiple times, but specifies different pNFS roles
 >    each time, the server might return different client IDs.  Given that
 >    different pNFS roles might have different client IDs, the client may
 >    ask for different properties for each role/client ID.

The second set of changes is in section 13.1, clarifying that the
server should have a preference to match the client's request.

15998c15998,16020
<    results, with each result indicating each role.
---
 >    results, with each result indicating each role.
 >
 >
 >    In the case of a server with concurrent PNFS roles that are served 
by a
 >    common client ID, if the EXCHANGE_ID request from the client has zero
 >    or a combination of the bits set in eia_flags, the server result 
should
 >    set bits which represent the higher of the acceptable combination of
 >    the server roles, with a preference to match the roles requested by
 >    the client. Thus if a client request has (EXCHGID4_FLAG_USE_NON_PNFS |
 >    EXCHGID4_FLAG_USE_PNFS_MDS | EXCHID4_FLAG_USE_PNFS_DS) flags set, and
 >    the server is both a metadata server and a data server, serving both
 >    the roles by a common client ID, the server should return
 >    with (EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS) set.
 >
 >
 >    In the case of a server that has multiple concurrent PNFS roles,
 >    each role served by a unique client ID, if the client specifies
 >    zero or a combination of roles in the request, the server results
 >    should return with only one of the performed roles, from the 
combination
 >    specified by the client request. If the role specified by the server
 >    result does not match the intended use by the client, the client
 >    should send the EXCHANGE_ID specifying just the interested PNFS role.
 >

Regards,
Mahesh

15998c15998,16020
<    results, with each result indicating each role.
---
>    results, with each result indicating each role. 
> 
> 
>    In the case of a server with concurrent PNFS roles that are served by a 
>    common client ID, if the EXCHANGE_ID request from the client has zero 
>    or a combination of the bits set in eia_flags, the server result should
>    set bits which represent the higher of the acceptable combination of 
>    the server roles, with a preference to match the roles requested by 
>    the client. Thus if a client request has (EXCHGID4_FLAG_USE_NON_PNFS |
>    EXCHGID4_FLAG_USE_PNFS_MDS | EXCHID4_FLAG_USE_PNFS_DS) flags set, and 
>    the server is both a metadata server and a data server, serving both 
>    the roles by a common client ID, the server should return 
>    with (EXCHGID4_FLAG_USE_PNFS_MDS | EXCHGID4_FLAG_USE_PNFS_DS) set.
> 
> 
>    In the case of a server that has multiple concurrent PNFS roles,
>    each role served by a unique client ID, if the client specifies 
>    zero or a combination of roles in the request, the server results
>    should return with only one of the performed roles, from the combination 
>    specified by the client request. If the role specified by the server
>    result does not match the intended use by the client, the client
>    should send the EXCHANGE_ID specifying just the interested PNFS role.
> 
26218,26226c26240,26249
<    convey roles the client ID is to be used for in a pNFS environment
<    Note that the same client owner/server owner pair can have multiple
<    roles.  Multiple roles can be associated with the same client ID or
<    with different client IDs.  Thus, if a client sends EXCHANGE_ID from
<    the same client owner to the same server owner multiple times, but
<    specifies different pNFS roles each time, the server might return
<    different client IDs.  Given that different pNFS roles might have
<    different client IDs, the client may ask for different properties for
<    each role/client ID.
---
>    convey roles the client ID is to be used for in a pNFS environment.
>    The server MUST set one of the acceptable combination of these bits
>    in eir_flags, as specified in Section 13.1. Note that the same 
>    client owner/server owner pair can have multiple roles. Multiple roles
>    can be associated with the same client ID or with different client IDs.
>    Thus, if a client sends EXCHANGE_ID from the same client owner to the 
>    same server owner multiple times, but specifies different pNFS roles 
>    each time, the server might return different client IDs.  Given that 
>    different pNFS roles might have different client IDs, the client may 
>    ask for different properties for each role/client ID.
_______________________________________________
nfsv4 mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Tue, 12 Feb 2008 06:11:11 -0800 (PST)

It is on my todo list to address this. I haven't had time to consider
it yet. Hopefully this week.

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Feb 2008 13:14:16 -0800 (PST)

--- Mahesh Siddheshwar <[email-masked]> wrote:

> in the request and server results. 

Attached are the diffs I'd like to check in.
Index: nfsv41_middle_file_layout.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_file_layout.xml,v
retrieving revision 1.41
diff -u -r1.41 nfsv41_middle_file_layout.xml
--- nfsv41_middle_file_layout.xml	29 Jan 2008 08:19:32 -0000	1.41
+++ nfsv41_middle_file_layout.xml	14 Feb 2008 21:10:36 -0000
@@ -82,6 +82,36 @@
   result indicating each role.
 
   </t>
+  <t>
+
+  In the case of a server with concurrent PNFS roles that
+  are served by a common client ID, if the EXCHANGE_ID
+  request from the client has zero or a combination of the
+  bits set in eia_flags, the server result should set bits
+  which represent the higher of the acceptable combination
+  of the server roles, with a preference to match the roles
+  requested by the client. Thus if a client request has
+  (EXCHGID4_FLAG_USE_NON_PNFS | EXCHGID4_FLAG_USE_PNFS_MDS
+  | EXCHID4_FLAG_USE_PNFS_DS) flags set, and the server
+  is both a metadata server and a data server, serving
+  both the roles by a common client ID, the server
+  SHOULD return with (EXCHGID4_FLAG_USE_PNFS_MDS |
+  EXCHGID4_FLAG_USE_PNFS_DS) set.
+
+  </t>
+  <t>
+
+  In the case of a server that has multiple concurrent
+  PNFS roles, each role served by a unique client ID,
+  if the client specifies zero or a combination of roles
+  in the request, the server results SHOULD return only
+  one of the roles from the combination specified by the
+  client request. If the role specified by the server
+  result does not match the intended use by the client,
+  the client should send the EXCHANGE_ID specifying just
+  the interested PNFS role.
+
+  </t>
 
   <t>
    If a pNFS metadata client gets a layout that refers it to an NFSv4.1
@@ -132,6 +162,24 @@
    a metadata server filehandle to cached data.
   </t>
 
+  <t>
+   If a server is both a metadata server and a data server, the server
+   might need to distinguish operations on files that directed to the metadata server from those that are directed to the data server. It is RECOMMENDED
+   that values of the filehandles returned by the
+   LAYOUTGET operation to be different
+   than the value of the filehandle returned by the OPEN of the same file.
+  </t>
+  <t>
+   Another secenario is for the metadata server and the
+   storage device to be distinct from one client's point of
+   view, and the roles reversed from another client's point
+   of view. For example, in the cluster file system model
+   a metadata server to one client, may be a data server to
+   another client. If NFSv4.1 is being used as the storage
+   protocol, then pNFS servers need to encode the values
+   of filehandles according to their specific roles.
+  </t>
+
  </section>
 
  <section title="File Layout Definitions" anchor="file_layout_definitions">
Index: nfsv41_middle_op_exchange_id.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_op_exchange_id.xml,v
retrieving revision 1.31
diff -u -r1.31 nfsv41_middle_op_exchange_id.xml
--- nfsv41_middle_op_exchange_id.xml	17 Dec 2007 22:17:39 -0000	1.31
+++ nfsv41_middle_op_exchange_id.xml	14 Feb 2008 21:10:36 -0000
@@ -342,16 +342,20 @@
      EXCHGID4_FLAG_USE_PNFS_MDS,  and
      EXCHGID4_FLAG_USE_PNFS_DS bits are described in <xref
      target="pnfs_session_stuff" /> and convey roles the
-     client ID is to be used for in a pNFS environment Note
-     that the same client owner/server owner pair can have
-     multiple roles. Multiple roles can be associated with
-     the same client ID or with different client IDs. Thus,
-     if a client sends EXCHANGE_ID from the same client
-     owner to the same server owner multiple times, but
-     specifies different pNFS roles each time, the server
-     might return different client IDs. Given that different
-     pNFS roles might have different client IDs, the client may
-     ask for different properties for each role/client ID.
+     client ID is to be used for in a pNFS environment.
+     The server MUST set one of the acceptable combinations
+     of these bits (roles) in eir_flags, as specified in <xref
+     target="pnfs_session_stuff" />.
+     Note that the same client owner/server owner pair can
+     have multiple roles. Multiple roles can be associated
+     with the same client ID or with different client
+     IDs. Thus, if a client sends EXCHANGE_ID from the
+     same client owner to the same server owner multiple
+     times, but specifies different pNFS roles each time,
+     the server might return different client IDs. Given
+     that different pNFS roles might have different client
+     IDs, the client may ask for different properties for
+     each role/client ID.
 
     </t>
 
Index: nfsv41_middle_pnfs.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_pnfs.xml,v
retrieving revision 1.70
diff -u -r1.70 nfsv41_middle_pnfs.xml
--- nfsv41_middle_pnfs.xml	13 Feb 2008 16:17:21 -0000	1.70
+++ nfsv41_middle_pnfs.xml	14 Feb 2008 21:10:39 -0000
@@ -1651,71 +1651,14 @@
   executing at any given time.
 </t>
 <t>
-  Various sub-cases can be distinguished.
+  Two sub-cases can be distinguished.
 
   <list style="numbers">
   <t>
-    The storage device uses NFSv4.1 as the storage protocol.  The same
+    The storage device uses NFSv4.1 as the storage protocol, i.e. same
     physical hardware is used to implement both a metadata and data
-    server. If an EXCHANGE_ID operation sent to the metadata server
-    has EXCHGID4_FLAG_USE_PNFS_MDS set and EXCHGID4_FLAG_USE_PNFS_DS
-    not set, the role of all sessions derived from the client ID is
-    metadata server-only.  If an EXCHANGE_ID operation sent to the
-    data server has EXCHGID4_FLAG_USE_PNFS_DS set and
-    EXCHGID4_FLAG_USE_PNFS_MDS not set, the role of all sessions
-    derived from the client ID is data server only.  These assertions
-    are true regardless whether the network addresses of the metadata
-    server and data server are the same or not.
-
-    <vspace blankLines='1' />
-
-    The client will use the same client owner for both the
-    metadata server EXCHANGE_ID and the data server
-    EXCHANGE_ID. Since the client sends one with
-    EXCHGID4_FLAG_USE_PNFS_MDS set, and the other with
-    EXCHGID4_FLAG_USE_PNFS_DS set, the server will need to
-    return unique client IDs, as well as server_owners, which
-    will eliminate ambiguity about dual roles the same
-    physical entity serves.
-  </t>
-
-  <t>
-    The metadata and data server each return EXCHANGE_ID
-    results with
-    EXCHGID4_FLAG_USE_PNFS_DS and EXCHGID4_FLAG_USE_PNFS_MDS both
-    set, the server_owner and server_scope results are the same,
-    and the client IDs are the same, and if RPCSEC_GSS is used,
-    the server principals are the same. As noted in <xref target="Trunking" /> the two servers are the same, whether they have the same
-    network address or not.  If the pNFS server
-    is ambiguous in its EXCHANGE_ID results as to what
-    role a client ID may be used for, yet still requires the NFSv4.1
-    request be directed in a manner specific to a role
-    (e.g. a READ request for a particular offset directed to
-    the metadata server role might use a different offset if the
-    READ was intended for the data server role, if the file
-    is using STRIPE4_DENSE packing,
-    see <xref target="sparse_dense" />),
-    the pNFS server may mark the
-    the metadata filehandle differently
-    from the data filehandle so that operations
-    addressed to the metadata server can be
-    distinguished from those directed to the data servers.
-    Marking the metadata and data server filehandles differently
-    (and this is RECOMMENDED) is possible because the former are
-    derived from OPEN operations, and the latter are derived
-    from LAYOUTGET operations.
-
-    <vspace blankLines='1' />
-
-    Note, that it may be the case that while the metadata server and
-    the storage device are distinct from one client's point of view,
-    the roles may be reversed according to another client's point of
-    view.  For example, in the cluster file system model a metadata
-    server to one client, may be a data server to another client.  If
-    NFSv4.1 is being used as the storage protocol, then pNFS servers
-    need to mark filehandles according to their specific roles.
-
-    <vspace blankLines='1' />
+    server. See <xref target="pnfs_session_stuff" />
+    for a description how multiple roles are handled.
   </t>
 
   <t>

---

From: Robert Gordon <[email-masked]>
Date: Thu, 14 Feb 2008 16:02:20 -0600

This also looks good with the following nit.
On Feb 14, 2008, at 3:14 PM, Mike Eisler wrote:

> +   might need to distinguish operations on files that directed to the

might need to distinguish operations on files directed to the

> +   metadata server from those that are directed to the data server.

--
Robert
"Always do your best"