Subject: [nfsv4] Some pending changes for draft-20

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 7 Feb 2008 13:07:44 -0500

Just want people to be aware of these changes that will be going 
into the CVS repository tomorrow (and will eventually be part of 
draft-20).  They are not major but since they are more than 
trivial, and we are in last-call, I'd like to make sure that
people get an adquate chance to see them.  

Although I will check this stuff in tomorrow, there is no need
to rush on issues/fixes/changes, since this stuff will be
easily updated even after it is checked into the CVS repository.  

Changes are:

    Deal with the issue of the interactions of delegations
    and the accesstime attribute.  (Bruce raised this).

    Some cleanup of reboot recovery, in particular dropping
    references to the obsolete error NFS4ERR_STALE_STATEID.

    Some expansion of the discussion of lease renewal, both 
    to explain issues related to interaction with lower 
    levels of software that might delay or obstruct messages 
    and also to better explain the connection between renewal 
    frequency and the grace period.

Index: nfsv41_middle_caching.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_caching.xml,v
retrieving revision 1.29
diff -c -r1.29 nfsv41_middle_caching.xml
*** nfsv41_middle_caching.xml	29 Jan 2008 08:19:32 -0000	1.29
--- nfsv41_middle_caching.xml	7 Feb 2008 16:30:28 -0000
***************
*** 649,661 ****
      </t>
      <t>
        When a client has a read open delegation, it is assured that
!       neither the contents, the attributes, nor the names of any
        links to the file will change without its knowledge, so long as
the
        delegation is held.  When a client has a write open delegation,
it
        may modify the file data locally since no other client will be 
        accessing the file's data.  The client holding a write
delegation 
        may only locally affect file attributes which are intimately 
!       connected with the file data: size, time_modify, change.
Changes
        to other attributes must be reflected on the server.
      </t>
      <t>
--- 649,663 ----
      </t>
      <t>
        When a client has a read open delegation, it is assured that
!       neither the contents, the attributes (with the exception of 
!       time_access), nor the names of any
        links to the file will change without its knowledge, so long as
the
        delegation is held.  When a client has a write open delegation,
it
        may modify the file data locally since no other client will be 
        accessing the file's data.  The client holding a write
delegation 
        may only locally affect file attributes which are intimately 
!       connected with the file data: size, change, time_access,
!       time_metadata, and time_modify, change.
        to other attributes must be reflected on the server.
      </t>
      <t>
Index: nfsv41_middle_state_mgmt.xml
===================================================================
RCS file: /cvs/stdk/NFSv41/nfsv41_middle_state_mgmt.xml,v
retrieving revision 1.23
diff -c -r1.23 nfsv41_middle_state_mgmt.xml
*** nfsv41_middle_state_mgmt.xml	29 Jan 2008 08:19:32 -0000
1.23
--- nfsv41_middle_state_mgmt.xml	7 Feb 2008 16:30:31 -0000
***************
*** 601,606 ****
--- 601,644 ----
  
      </t>
      <t>
+       Because a lease may expire if is ever the case that 
+       the lease period goes by without a SEQUENCE operation
+       being received by the server, the client, if it is
+       not sending COMPOUNDs for other reasons, should
+       send COMPOUNDs in order to to ensure that the lease
+       is renewed.  A COMPOUND consisting only of a SEQUENCE
+       operation is sufficient for this purpose.  Because of 
+       the possiblity that lease expiration will result in a
+       loss of locking state, care needs to be exercised by
+       the client to reduce the probability that communications
+       issues will avoidably allow lease lease expiration:
+       <list style='symbols'>
+         <t>
+           When trunking is in effect, the client should 
+           consider issuing multiple requests on different
+           connections, in order to ensure that renewal
+           occurs, even in the event of blockage in the 
+           path used for one of those connections.
+         </t>
+         <t>
+           The effect of lower levels of the software stack
+           needs to be carefully considered,  For example,
+           if the remote side of TCP connection being 
+           unresponsive can result in a retransmission backoff
+           such that successive attempts to resend the request 
+           have an interval approaching or exceeding the lease
+           time, the result may be lease expiration, or a grace
+           period that expires on the server before the client
+           is made aware of a server reboot (See 
+           <xref target="reclaim_locks" />).  To avoid such
+           situations, clients may need to take special actions
+           to ensure that new attempts to contact the server 
+           happen with sufficient frequency  that leases and
+           grace period do not expire unnecessarily.
+         </t>
+       </list>
+     </t>
+     <t>
        If the server renews the lease upon receiving
        a SEQUENCE operation, the server MUST NOT allow the lease
        to expire while the rest of the operations
***************
*** 683,693 ****
        protecting the READ and WRITE operations. Any that reach the
        server before the server can safely determine that the client
        has recovered enough locking state to be sure that such
!       operations can be safely processed must be rejected, either
!       because the state presented is no longer valid
!       (NFS4ERR_STALE_CLIENTID or NFS4ERR_STALE_STATEID) or because
!       subsequent recovery of locks may make execution of the operation
!       inappropriate (NFS4ERR_GRACE).
      </t>
      <section title="Client Failure and Recovery" >
        <t>
--- 721,742 ----
        protecting the READ and WRITE operations. Any that reach the
        server before the server can safely determine that the client
        has recovered enough locking state to be sure that such
!       operations can be safely processed must be rejected.
!       This will happen because either:
!       <list style='symbols'>
!         <t>
!           The state presented is no longer valid since it is 
!           associated with a now invalid clientid.  In this case the
!           client will receive either an NFS4ERR_BADSESSION or
!           NFS4ERR_DEADSESSION error and any attempt to attach a new
!           session to the existing clientid will encounter an
!           NFS4ERR_STALE_CLIENTID error.
!         </t>
!         <t>
!           Subsequent recovery of locks may make execution of the 
!           operation inappropriate (NFS4ERR_GRACE).
!         </t>
!       </list>
      </t>
      <section title="Client Failure and Recovery" >
        <t>
***************
*** 790,796 ****
          When state information and the associated locks are lost
          as a result of a server reboot, the protocol must provide
          a way to cause that state to be re-established.  The 
!         approach used is to define, for most type of locking
          state (layouts are an exception), a request whose function 
          is to allow the client to 
          re-establish on the server a lock first obtained from a
--- 839,845 ----
          When state information and the associated locks are lost
          as a result of a server reboot, the protocol must provide
          a way to cause that state to be re-established.  The 
!         approach used is to define, for most types of locking
          state (layouts are an exception), a request whose function 
          is to allow the client to 
          re-establish on the server a lock first obtained from a
***************
*** 810,816 ****
          sessions are handled normally, but locking requests are
          subject to special restrictions.  Only 
          reclaim-type locking requests are allowed, unless the
!         server is able to< reliably determine (through state
          persistently maintained across reboot instances), that
          granting any such lock cannot possibly conflict with a
          subsequent reclaim.  
--- 859,865 ----
          sessions are handled normally, but locking requests are
          subject to special restrictions.  Only 
          reclaim-type locking requests are allowed, unless the
!         server is able to reliably determine (through state
          persistently maintained across reboot instances), that
          granting any such lock cannot possibly conflict with a
          subsequent reclaim.  
***************
*** 859,867 ****
          done a global RECLAIM_COMPLETE.  The server SHOULD NOT
terminate the
          grace period before a time equal to the lease period in order
          to give clients an opportunity to find out about the server 
!         reboot.  Some additional time in order to allow time to 
          establish a new client ID and session and to effect lock 
!         reclaims may be added.  Note that analogous rules apply to
          file system-specific grace periods discussed in
          <xref target="transition_state" />.
        </t>
--- 908,926 ----
          done a global RECLAIM_COMPLETE.  The server SHOULD NOT
terminate the
          grace period before a time equal to the lease period in order
          to give clients an opportunity to find out about the server 
!         reboot, as a result of issuing requests on associated 
!         sessions with a frequency governed by the lease time.  
!         Note that when a client does not issue such requests (or they
!         are issued by the client but not received by the server),
!         it is possible for the grace period to expire before the
client
!         finds out that the server reboot has occurred.
!       </t>
!       <t>
!         Some additional time in 
!         order to allow time to 
          establish a new client ID and session and to effect lock 
!         reclaims may be added to the lease time.  Note that 
!         analogous rules apply to
          file system-specific grace periods discussed in
          <xref target="transition_state" />.
        </t>
***************
*** 1242,1251 ****
        reboot or restart.  Note that this includes situations
        in which sessions are persistent and locking state is
        lost.  In this class of instances, the client will
!       receive an error (NFS4ERR_STALE_STATEID on an
!       operation that takes a stateid as an argument or
!       NFS4ERR_STALE_CLIENTID on an operation that takes a
!       sessionid or client ID) and the client will proceed
        with normal crash recovery as described in the <xref
        target="reclaim_locks" />.
      </t>
--- 1301,1310 ----
        reboot or restart.  Note that this includes situations
        in which sessions are persistent and locking state is
        lost.  In this class of instances, the client will
!       receive an error (NFS4ERR_STALE_CLIENTID on an 
!       operation that takes client ID, usually as part of 
!       recovery in response to a problem with the current 
!       session) and the client will proceed
        with normal crash recovery as described in the <xref
        target="reclaim_locks" />.
      </t>
_______________________________________________
nfsv4 mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 7 Feb 2008 14:58:55 -0500

> Index: nfsv41_middle_caching.xml
			 ^^^^^^^^^^^^^^^^^^^^^^^

Move the "and" to just before "change"?

> Index: nfsv41_middle_state_mgmt.xml
				      ^^^
				     if it is ever...

(But actually I'd just remove "is ever the case that" entirely and make
it something like "if a lease period ever goes by without...")

> +       issues will avoidably allow lease lease expiration:

This section already states when the server may revoke state in previous
and subsequent paragraphs.  I think the only new information here is the
suggestion of using a lone SEQUENCE operation, and the admonition to
take into account communication delays.  So how about replacing the
above by just:

	Absent other activity that would renew the lease, a COMPOUND
	consisting of a single SEQUENCE operation will suffice.  The
	client should also take into account communication delays.

The following is a little wordy:

> +           <xref target="reclaim_locks" />).

How about:

	TCP retransmission delays could approach the lease time.  This
	may be particularly likely when the server is unresponsive due
	to a reboot; see <xref target="reclaim_locks" />

		To avoid such
> +           grace period do not expire unnecessarily.

The above seems redundant given what's just been said.

>         to expire while the rest of the operations

I think a comma before this "and" would help the reader.

>       <section title="Client Failure and Recovery" >

That's all I saw.  Thanks for the updates.

--b.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 7 Feb 2008 17:01:06 -0500

> Move the "and" to just before "change"?

Actually "change" appears twice in the list.  The second "change" should
just go.

> (But actually I'd just remove "is ever the case 
> that" entirely and make it something like "if a 
> lease period ever goes by without...")

OK.

> This section already states when the server may revoke 
> state in previous and subsequent paragraphs.  I think 
> the only new information here is the suggestion of 
> using a lone SEQUENCE operation, and the admonition to
> take into account communication delays.  So how about 
> replacing the above by just:

>	client should also take into account communication delays.

"communication delays" may suggest something too limited.
How about "communication-related delays"?

>	to a reboot; see <xref target="reclaim_locks" />

This version seems to idly observe something, rather than being a
call to action or at least the verification of something.

So I've added stuff to the introductory paragraph to address that.

How about:

    <t>
      Absent other activity that would renew the lease, a COMPOUND
      consisting of a single SEQUENCE operation will suffice.  The
      client should also take into account communication-related
      delays, and take steps to ensure that renewal messages
      actually reach the server in good time.  For example:
      <list style='symbols'>
        <t>
          When trunking is in effect, the client should 
          consider issuing multiple requests on different
          connections, in order to ensure that renewal
          occurs, even in the event of blockage in the 
          path used for one of those connections.
        </t>
        <t>
	    TCP retransmission delays could approach the lease 
          time.  This may be particularly likely when the 
          server is unresponsive due to a reboot; 
          see <xref target="reclaim_locks" />
        </t>
      </list>
    </t>

> I think a comma before this "and" would help the reader.

OK.

Thanks.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Thursday, February 07, 2008 2:59 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Some pending changes for draft-20

			 ^^^^^^^^^^^^^^^^^^^^^^^

				      ^^^
				     if it is ever...

How about:

		To avoid such

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 7 Feb 2008 17:25:51 -0500

> Actually "change" appears twice in the list.  The second "change" should

Pfft, shows how carefully I was reading!  Thanks.

> call to action or at least the verification of something.

Fair enough.

>       actually reach the server in good time.  For example:

Looks good.  Thanks!

>           time.

I had one small second thought here: on rereading I confused myself for
a moment by reading "the lease time" as a point in time rather than a
duration.  So, maybe make the above sentence the somewhat more redundant
but completely unambiguous:

	TCP retransmission delays might become so large as to approach
	the length of the lease period.

??--b.

>	    This may be particularly likely when the

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 8 Feb 2008 09:20:35 -0500

> So, maybe make the above sentence the somewhat more redundant

I was thinking:

      TCP retransmission delays might become so large as to 
      approach or exceed the length of the lease period.

That would make it completely and totally unambiguous, even if somewhat
more redundant :-)

Thanks.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Thursday, February 07, 2008 5:26 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Some pending changes for draft-20

Fair enough.

Looks good.  Thanks!

??--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 8 Feb 2008 09:34:32 -0500

> That would make it completely and totally unambiguous, even if somewhat

OK!  Sounds good to me.

--b.

---

From: Peter Staubach <[email-masked]>
Date: Fri, 08 Feb 2008 09:59:44 -0500

>       actually reach the server in good time.  For example:

It seems to me that this may read more smoothly as:

"The client should also take communication-related delays into
account and take steps to ensure that the renewal messages actually
reach the server in good time."

    Thanx...

       ps

>>         links to the file will change without its knowledge, so long
>>     
> as
>   
>> the
>>         delegation is held.  When a client has a write open
>>     
> delegation,
>   
>> it
>>         links to the file will change without its knowledge, so long
>>     
> as
>   
>> the
>>         delegation is held.  When a client has a write open
>>     
> delegation,
>   
>> it
>> !       subsequent recovery of locks may make execution of the
>>     
> operation
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 8 Feb 2008 10:04:26 -0500

OK.

Thanks.

-----Original Message-----
From: Peter Staubach [mailto:[email-masked]] 
Sent: Friday, February 08, 2008 10:00 AM
To: Noveck, Dave
Cc: J. Bruce Fields; [email-masked]
Subject: Re: [nfsv4] Some pending changes for draft-20

    Thanx...

       ps