Subject: Re: [nfsv4] NFSv4.2 inter-server question

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 3 Mar 2015 17:42:34 -0500

On Fri, Feb 27, 2015 at 11:42 AM, Tom Haynes
<[email-masked]> wrote:
>
> On Feb 24, 2015, at 1:12 PM, Olga Kornievskaia <[email-masked]> wrote:
>
> Hi Tom,
>
> Minor typos:
> 1. I noticed that section 4.2.2, 3rd paragraph, has COPY_NOTICE instead of
> COPY_NOTIFY.
>
>
> Thanks, fixed!
>
>
> 2. In 15.3.3, there is a mention of COPY_REVOKE, I'm assuming that it's
> OFFLOAD_CANCEL?
>
>
> Ditto
>
>
> Question:
> In 15.2.3, 6th paragraph, it says:
>
>   For an intra-server copy, both the ca_src_stateid and ca_dst_stateid
>   MUST refer to either open or locking states provided earlier by the
>   server.
>
>
> Can this be a delegation stateid?  Or it must be open stateid or locking
> stateid? (my question is for intra-server copy as well).
>
>
>
> I believe the thought process was that the client had to get a lock request
> in order to provide for the situation below. And according to Section 8.2.5
> of 5661, the lock stateid would take precedence.
>
> But yes, if the client had established a delegation, then according to that
> same Section, a delegation stateid should be used.
>
> Any dissent?
>
> If not, I’ll make a clarification to the text:
>
>    For an intra-server copy, both the ca_src_stateid and ca_dst_stateid
>    MUST refer to either delegation, locking, or open provided earlier by the
>    server (See Section 4.4.1). The order of selection is explained in
> Section
>    8.2.5 of [RFC5661].

Yes, thank you for adding delegation to the list. It makes things clear.

>
>
>
>
>
> Also in general, if a client has a delegation for the source file it's
> copying and allowed to use it for the copy, should there be some guidance in
> the spec regarding dealing with recall of that delegation. Does the client
> must wait until the copy completes before returning the delegation? I
> imagine that otherwise, the destination server that uses that stateid in the
> READ can fail with a BAD_STATEID error.
>
>
>
>
> This is expected behavior and why we want the destination to have the
> stateid.

Again for the sake of clarity, I think it would be great to add a
statement saying: Upon receiving a delegation recall, a client MUST
wait until the copy completes before turning the delegation.  Just
like RFC 3530 and 5661 explicitly state that all writes must be
flushed before delegation is returned, this new operation COPY is just
like IO and needs to be completed.

>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Tom Haynes <[email-masked]>
Date: Tue, 3 Mar 2015 15:24:46 -0800

> On Mar 3, 2015, at 2:42 PM, Olga Kornievskaia <[email-masked]> wrote:
> 
> <[email-masked] <mailto:[email-masked]>> wrote:
>> 

I do not think the intent in 5661 is to allow the client to complete all I/O to
the server for the delegation being recalled.

   On recall, the client holding the delegation needs to flush modified
   state (such as modified data) to the server and return the
   delegation.  The conflicting request will not be acted on until the
   recall is complete.  The recall is considered complete when the
   client returns the delegation or the server times its wait for the
   delegation to be returned and revokes the delegation as a result of
   the timeout.  In the interim, the server will either delay responding
   to conflicting requests or respond to them with NFS4ERR_DELAY.
   Following the resolution of the recall, the server has the
   information necessary to grant or deny the second client's request.

   At the time the client receives a delegation recall, it may have
   substantial state that needs to be flushed to the server.  Therefore,
   the server should allow sufficient time for the delegation to be
   returned since it may involve numerous RPCs to the server.  If the
   server is able to determine that the client is diligently flushing
   state to the server as a result of the recall, the server may extend
   the usual time allowed for a recall.  However, the time allowed for
   recall completion should not be unbounded.

If the COPY is a 1TB file and the recall happens 1MB in, I’d say that
is unbounded.

The client can probe the destination to see if it is close to being done.
If it is, then it can hold off. If it is not, then it has to abandon the COPY
and it can retry as needed.

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 3 Mar 2015 19:18:41 -0500

> and it can retry as needed.

In that case, I don't see a reason for the client to ever use a
delegation stateid because if it does it can get penalized for it. By
that I mean, when the server decides to revoke the delegation, the
client has to fail the copy by sending the OFFLOAD_CANCEL to the
source server, leading to the destination server to receive an error
on READ which it would propagate to the client, and then the client
would restart the copy with the open stateid instead. So why ever
gamble on this overhead when you can always just use the open_stated?
So then perhaps a wording that says, delegation should not be used be
a better clarification.

---

From: Rick Macklem <[email-masked]>
Date: Tue, 3 Mar 2015 20:20:53 -0500 (EST)

Tom Haynes wrote:
> 
> 
> 
> 
> 
> 
> On Mar 3, 2015, at 2:42 PM, Olga Kornievskaia < [email-masked] >
> wrote:
> 
> < [email-masked] > wrote:
> 
> 
> 
> On Feb 24, 2015, at 1:12 PM, Olga Kornievskaia < [email-masked] >
> wrote:
> 
> 
> 
I agree with Tom on this. When a delegation is recalled, a client can
acquire Open(s) against the server and use those stateids for I/O.
(It will either have to wait for I/O RPCs in progress to complete or
 be prepared to retry them with an Open stateid instead of the delegation
 stateid.)

> recall completion should not be unbounded.
The state that needs to be flushed are the byte range locks and opens
that were issued locally on the client for the delegation. This shouldn't
normally take very long, although I agree that the RFCs don't set a hard
upper limit on the delay time. (It would be nice if the RFCs specified
actual time limits for such things as: "how long can the server delay
the reply for a blocking lock" or "how long can the server wait before
issuing a NFS4ERR_DELAY reply to an operation" but it does not. It seems
to me the upper bound should be in the millisecond range, but it would
be interesting to hear other opinions?)

rick