Subject: [nfsv4] Additional Comments on NFS4.1 I-D 18 Section 12 Parallel NFS (pNFS)

From: [email-masked]
Date: Thu, 10 Jan 2008 12:26:03 -0500

I have some more comments....

12.5.5.1
	It may be useful for clients to "forget" details about what	
	layouts and ranges the client actually has, leading to the	
	server's layout ranges being beyond those what the client
"thinks"	
	it has. As long as the client does not assume it has layouts
that	
	are beyond what the server has granted, this is a safe practice.

	Regardless, when a client forgets what ranges and layouts it
has,	
	and it gets a CB_LAYOUTRECALL recall and is not certain whether
it	
	has a layout for the range specified when the server sends a	
	CB_LAYOUTRECALL, the client MUST follow up with a LAYOUTRETURN
for	
	what the server asked for. If the client is partially forgetting

	and partially remembering, and it is certain it does not have
the	
	range being recalled, it MUST return NFS4ERR_NOMATCHING_LAYOUT.

s/gets a CB_LAYOUTRECALL recall/receives a CB_LAYOUTRECALL operation/
s/for what the server asked for/for what the server recalled./

I am confused by the notion of a client that "is not certain".  A client
can "forget" information, but that does not imply that the client "is
not certain" about what layout state it has.  If the layout is
forgotten, then the client knows nothing about it.

Perhaps the intention is to say that "the client is not certain if the
server thinks the client has been granted the layout".  But that
confusing sentence adds nothing, because the server would not issue a
recall unless it thinks the client has the layout.

So, I think that the above paragraph should be simplified as follows:

	It may be useful for clients to "forget" details about what	
	layouts and ranges the client actually has, leading to the	
	server's layout ranges being beyond those what the client
"thinks"	
	it has. As long as the client does not assume it has layouts
that	
	are beyond what the server has granted, this is a safe practice.

	When a client forgets what ranges and layouts it has,	
	and it receives a CB_LAYOUTRECALL operation, the client MUST
follow up
      with a LAYOUTRETURN for what the server recalled.

That was my recollection of our discussion at the 70th IETF.

12.5.5.2.1.3
s/Consider the race the metadata/Consider the race from the metadata/

12.5.5.2.1.4
  "In the event the server
   sends a bulk recall while the client has pending or in progress
   LAYOUTRETURN, CB_LAYOUTRECALL, or LAYOUTGET, the client returns
   NFS4ERR_DELAY.  In the event the client sends a LAYOUTGET or
   LAYOUTRETURN while a bulk recall is in progress, the server returns
   NFS4ERR_RECALLCONFLICT."

When a server receives a response of NFS4ERR_DELAY, is the bulk recall
to be considered "in progress"?

If not are we guaranteed forward progress?  If a client has multiple
threads writing to multiple files, is it possible that there is always
one LAYOUTGET operation outstanding at precisely the instant that the
client receives the CB_LAYOUTRECALL?  The client will respond with
NFS4ERR_DELAY.  The server will presumably back off and retry too.
There appears to be no guidance on client handling of
NFS4ERR_RECALLCONFLICT errors.  If the client waits to retry, the client
and server might both retry at the same frequency.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Thu, 10 Jan 2008 09:56:54 -0800 (PST)

> I have some more comments....
[...]

I agree with all of Jason's comments, and have made the changes
to the i-d's source text.

Excellent point.

I proposed adding this sentence to the end of the paragraph:

``  If the client sends a LAYOUTGET or LAYOUTRETURN after
 the server receives NFS4ERR_DELAY from a bulk recall,
 then to ensure forward progess, the server MAY return
 NFS4ERR_RECALLCONFLICT. ''

---

From: Benny Halevy <[email-masked]>
Date: Thu, 10 Jan 2008 19:57:05 +0200

>       with a LAYOUTRETURN for what the server recalled.

I like the rephrased paragraph, yet one more detail is still missing
from it:

the client MUST follow up with a LAYOUTRETURN for what the server recalled
or, alternatively, return the NFS4ERR_NOMATCHING_LAYOUT error if it 
has no layout to return in the recalled range.

> That was my recollection of our discussion at the 70th IETF.
The client should avoid this "live lock" scenario e.g. by recording the fact
it received a bulk recall and refrain from issuing any more layout gets and
returns to the recalled fsids until it gets the callback again, processes,
and completes it.

If it did not get it in a reasonable time frame it can either attempt
resume of normal operation, guessing that the server might have changed
his mind, however, this can lead to the scenario you mentioned above,
so if this recovery strategy fails and the client still gets
NFS4ERR_RECALLCONFLICT it can use the nuke option and tear down all
sessions to explicitly invalidate its state and start over from
scratch.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Thu, 10 Jan 2008 20:08:31 +0200

> returns to the recalled fsids until it gets the callback again, processes,

Actually, once all outstanding layout gets and returns to the recalled
fsid(s) complete, either successfully or with an error, and if the client
has not received the CB_LAYOUTRECALL again it should just send a LAYOUTRETURN
for the recalled FSID (or ALL), voluntarily and proactively before
trying to get any new layout for these fsid(s).

---

From: Mike Eisler <[email-masked]>
Date: Thu, 10 Jan 2008 10:13:56 -0800 (PST)

> has no layout to return in the recalled range.

Even better. Done.

> returns to the recalled fsids until it gets the callback again,

While I think the above approach is a good optimization, I
think Jason's solution is more general and simpler. 
Jason's approach puts the onus on the server, which is 
obviously in the best position to know whether a previously
attempted bulk recall is in progress.

---

From: [email-masked]
Date: Fri, 11 Jan 2008 12:33:45 -0500

I agree with Mike.  We should add his proposed text to the spec.
Benny's solution makes sense, but I'm not sure it needs to be part of
the spec.

-Jason

--- Mike Eisler <[email-masked]> wrote:

> ``  If the client sends a LAYOUTGET or LAYOUTRETURN after  the server
> receives NFS4ERR_DELAY from a bulk recall,  then to ensure forward
> progess, the server MAY return  NFS4ERR_RECALLCONFLICT. ''

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Thursday, January 10, 2008 1:14 PM
To: [email-masked]
Subject: Re: [nfsv4] Additional Comments on NFS4.1 I-D 18 Section 12
ParallelNFS (pNFS)

Even better. Done.

---

From: Benny Halevy <[email-masked]>
Date: Sat, 12 Jan 2008 10:46:37 +0200

> Benny's solution makes sense, but I'm not sure it needs to be part of
It doesn't have to be mandated on the client so it can be provided as an 
implementation recommendation.
The point is that the client received the callback so it SHOULD return 
the recalled layouts ASAP
even if it returned a DELAY error (and if it returned success then it 
MUST rather than SHOULD...).

Benny
>>>    LAYOUTRETURN while a bulk recall is in progress, the server
>>>       
>> returns
>>     
>>>    NFS4ERR_RECALLCONFLICT."
>>>
>>> When a server receives a response of NFS4ERR_DELAY, is the bulk
>>>       
>> recall
>>     
>>> If not are we guaranteed forward progress?  If a client has
>>>       
>> multiple
>>     
>>> threads writing to multiple files, is it possible that there is
>>>       
>> always
>>     
>>> one LAYOUTGET operation outstanding at precisely the instant that
>>>       
>> the
>>     
>>> NFS4ERR_RECALLCONFLICT errors.  If the client waits to retry, the
>>>       
>> client
>>