Subject: [nfsv4] pnfs session issue

From: Mike Eisler <[email-masked]>
Date: Wed, 20 Feb 2008 09:10:46 -0800 (PST)

Today at the Austin bake-a-thon a question came up about whether the
client needs to issue SEQUENCE operations to a data server in order
to keep the session alive.

My answer is yes.

A related question then is, how often must the SEQUENCE be issued?

My answer is within the "lease_time" attribute.

The next question was where does the client get the lease_time
attribute?

Here the answer is not very easy, because a data server could be
providing data servers for multiple MDSes, each of which with
different lease_time attributes.

I don't want to add GETATTR to the list of operations data servers
have to support.

Ideas?

_______________________________________________
nfsv4 mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 20 Feb 2008 11:18:01 -0600

> I don't want to add GETATTR to the list of operations data servers

The lease period of the MDS.

I don't remember the specifics at the moment but the DS is
suppose to stop servicing read/writes at some point for a client
that has stopped renewing it lease with the MDS given that the
stateid is from the MDS.  This presumes that there is some
knowledge between the DS and MDS about an acceptable amount
of time that the DS can continue to service that request
and therefore part of the control protocol...

Yes, additional burden for the control protocol but seems
to be a straightforward method of explaining and managing
lease periods for the pNFS server as a whole.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 20 Feb 2008 13:16:46 -0500

> The lease period of the MDS.

As Mike points out, there may be multiple MDS's.

For me, the issue is that that the period for DS is likely to higher
(you want it to be higher) than for the MDS's.  The MDS owns the locks
and so if you make the lease time large (e.g. ten minutes), it may be a
long time before somebody else waiting for a conflicting lock will get
his lock given that your lease will take a long time to expire.  

On the DS, that issue doesn't arise.  Instead, you need some time after
which you decide that you don't have to keep around session state if the
client is long gone.  No pressing need to expire it and if communication
issues might persist a while, you don't want to kill the session and
give up the benefits of EOS lightly.

Maybe we can say that the lease period for the DS must never be shorter
than the lease period for any of the associated MDS's.  This would mean
that the client would not have a way of getting the DS lease period but
would have a way of gettng a value guaranteed to be no larger than the
DS lease period.  That would prevent session loss.  The only issue would
be whether it might be too short in the sense that it would increase the
lease renewal overhead (and the client might have lots of DS's to
signal).

If using the MDS lease time in this fashion is not good enough, then the
choices would seem to be GETATTR (which Mike doesn't like for good
reason), and having this returned in CREATE_SSSIONresok, which is an XDR
change that we'd like to avoid at this stage. 

I guess one other option is to say the time must be no shorter than the
lease period for any of the MDS's and also no shorter than some
spec-defined minimum value (e.g. five minutes).

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Wednesday, February 20, 2008 12:18 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] pnfs session issue

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 20 Feb 2008 12:36:40 -0600

> DS lease period.  That would prevent session loss.  The only issue  

I believe this was the consensus in the room in an attempt to
interpret my previous response.  DS lease period will be no shorter
than the MDS that it is associated with and if there are multiple MDS
then it is at least the maximum of all MDS lease periods.

> lease renewal overhead (and the client might have lots of DS's to

I don't see this as a concern in that the client has overhead in
doing business with a pNFS server and this is just one of the things
it needs to account for and track; isn't any more burdensome than
doing read/write traffic to a DS. :-)

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Thu, 21 Feb 2008 11:09:03 -0800 (PST)

> interpret my previous response.  DS lease period will be no shorter

Note that the science of interpreting Spencer's responses has been
termed: "Spencerianism". 

> doing read/write traffic to a DS. :-)

Attached are diffs.

   -mre
    Arch-Spencerianist
Index: nfsv41_middle_core_infrastructure.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_core_infrastructure.xml,v
retrieving revision 1.86
diff -u -w -b -r1.86 nfsv41_middle_core_infrastructure.xml
--- nfsv41_middle_core_infrastructure.xml	21 Feb 2008 12:58:17 -0000	1.86
+++ nfsv41_middle_core_infrastructure.xml	21 Feb 2008 19:05:08 -0000
@@ -3453,7 +3453,7 @@
    </t>
    </section> <!-- Obligations of the Server -->
 
-   <section anchor="Obligations of the Client" title="Obligations of the Client">
+   <section anchor="Obligations_of_the_Client" title="Obligations of the Client">
    <t>
    The client SHOULD honor the following obligations in order to
    utilize the session:
@@ -3464,7 +3464,10 @@
      sending operations risks having the session be destroyed
      by the server. This is because sessions consume
      resources, and resource limitations may force the
-     server to cull an inactive session.
+     server to cull an inactive session. If the interval between
+     two SEQUENCE operations each sent on different COMPOUND requests is less
+     than the lease_time attribute (<xref target="attrdef_lease_time" />),
+     then the session is considered active.
    </t>
    <t>
      Destroy the session when not needed. If a client has
Index: nfsv41_middle_file_layout.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_file_layout.xml,v
retrieving revision 1.47
diff -u -w -b -r1.47 nfsv41_middle_file_layout.xml
--- nfsv41_middle_file_layout.xml	21 Feb 2008 12:58:17 -0000	1.47
+++ nfsv41_middle_file_layout.xml	21 Feb 2008 19:05:08 -0000
@@ -176,12 +176,49 @@
    Another scenario is for the metadata server and the
    storage device to be distinct from one client's point of
    view, and the roles reversed from another client's point
-   of view. For example, in the cluster file system model
-   a metadata server to one client, may be a data server to
+   of view. For example, in the cluster file system model,
+   a metadata server to one client may be a data server to
    another client. If NFSv4.1 is being used as the storage
    protocol, then pNFS servers need to encode the values
    of filehandles according to their specific roles.
   </t>
+  <section anchor="dsonly" title="Sessions Considerations for Data Servers">
+  <t>
+
+   <xref target="Obligations_of_the_Client" /> states
+   that a client has to keep its lease renewed in
+   order to prevent a session from being deleted by
+   the server. If the reply to EXCHANGE_ID has just the
+   EXCHGID4_FLAG_USE_PNFS_DS role set, then as noted in
+   <xref target="ds_ops"/> the client will not be able
+   to determine the data server's lease_time attribute,
+   because GETATTR will not be permitted. Instead, the
+   rule is that any time a client receives a layout
+   referring it to a data server that returns just
+   the EXCHGID4_FLAG_USE_PNFS_DS role, the client MAY
+   assume that the lease_time attribute from the metadata
+   server that returned the layout applies to the data
+   server. Thus the data server MUST be aware of the values
+   of all lease_time attributes of all metadata servers it
+   is providing I/O for, and MUST use the maximum of all such
+   lease_time values as the lease interval for all client
+   IDs and sessions established on it.
+
+  </t>
+  <t>
+
+   For example, if one metadata server has a lease_time
+   attribute of 20 seconds, and a second metadata
+   server has a lease_time attribute of 10 seconds,
+   then if both servers return layouts that refer to an
+   EXCHGID4_FLAG_USE_PNFS_DS-only data server, the data
+   server MUST renew a client's lease if the interval
+   between two SEQUENCE operations on different COMPOUND
+   requests is less than 20 seconds.
+
+  </t>
+ 
+  </section>
 
  </section>