Subject: RE: [nfsv4] pnfs: keeping the client and server layouts in sync

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 27 Jun 2006 08:44:15 -0400

I'm a little confused about your remarks and how they relate to the
ongoing attempt to close on outstanding issues, specifically the
pNFS and Sessions issue for which David Black has recently sent 
text.  The sessions use in that text referred to the handling of 
layout recall.

I'd like to get this cleared up since that issue has now been 
been put into status "editing", based on the understanding that
we had already reached working group consensus on it.  If you 
feel that that is not the case, we need to understand what the 
problem is.

Do your worries about using sessions have any effect on this text,
or are your worries about using sessions limited to a potential
future use of sessions to help (efficiently) enforce serialization 
in the case of voluntary layout returns?  If that's the case, then 
we have no problem (yet), since so far nobody has proposed doing 
what you are worried about. 

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Tuesday, June 27, 2006 4:59 AM
To: [email-masked]; Noveck, Dave; David Black; Brent Welch
Cc: [email-masked]
Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in sync

Dave, I'll try to summarize my observations here, more discussion
below...

LAYOUTGETs and RETURNs modify the layout state on both sides of the
wire.

The server owns the authoritative state and the changes to the state
flow from the server to the client via respective replies to GETs and
RETURNs.

To stay in sync they need to be serialized and applied in exactly the
same order on both sides of the wire.

The client can implement a simple serialization scheme by blocking any
GET or RETURN on possibly conflicting outstanding RETURNs or GETs.

A special case is CB_LAYOUTRECALL in which the client can assume and
avoid a conflict based on the recalled range rather than on any specific
LAYOUTRETURN.

Deadlock prevention can be done on the server by rejecting GETs that are
possibly conflicting with RECALLs the server sends until the server gets
the reply for that CB_LAYOUTRECALL.  Sessions can probably be used here
to fine tune this.

It might be possible to utilize sessions to serialize GETs and RETURNs
but the server will have to strictly order these operations so that the
client will know exactly which outstanding LAYOUTGETs were affected by a
RETURN it sent, and even then the client will probably have to guess
what the effect was or "force" a state sync by returning these layouts
after it got them if it's not sure how to resolve the conflict.

I think that trying to optimize potential performance issues using
sessions is going to be hard and I wouldn't want to risk correctness
while we're working on the problem. My gut feeling is that to be on the
safe side we should start with the simple model that does not rely on
sessions (and prevent deadlocks as described above) and then work out a
solution that uses sessions to optimize performance.

Benny

Noveck, Dave wrote:
> I think you raise some good points but I also disagree with your 
> analysis in part.
> 
> First of all, I agree that the current situation with LAYOUTRETURN is 
> worrisome, even apart from issues specific to recall.  I think there 
> just seem to be too many ways that the client and server can get out 
> of sync.  The text you quote "In order to ensure client/server 
> convergence on the layout state" makes me especially nervous since 
> convergence would only be needed if there a previous divergence to 
> converge from, and it isn't really clear how we can both allow that 
> divergence while guaranteeing later convergence.

agreed.

> 
> The important point is that the range of possible conflicts when you 
> have outstanding LAYOUTGET's may be larger than you might expect and 
> this has important consequence which I'll talk about later.
> 
> I don't agree that the issue becomes worse when recalling fsid's.
> It is true that there may be more conflicts in this case but all of 
> the conflicts are knowable by the client he has issued the LAYOUTGET.
> Since a recall of an FSID recalls the entire range file there is no 
> question of a hidden conflict where the range you get is bigger than 
> what you ask for.  In this case, you should not return a layout while 
> you have a conflicting LAYOUTGET in the pipe.  You simply have to wait

> to resolve the LAYOUTGET before doing the LAYOUTRETURN.  If you don't,

> the client's view and the servers view of things are likely to 
> diverge.
> 
> Now in the case of a LAYOUTRETURN for a small range, we still have the

> same issue.  Any LAYOUTGET, as you note, represents a potential 
> conflict even if the range sought does not overlap the range to be
returned.
> So it looks like the only way that you can keep the server and client 
> in sync is to say you can't do the LAYOUTRETURN when any LAYOUTGET for

> the same file is outstanding.

This is important.

> I think this is workable in the
> case of voluntary return but poses problems in the case of recall, but

> that sessions, provides a way out of the problem.  Unlike you, I think

> you really need sessions to get around some of the recall issues.
> 
> To see why a wait-until-GET-is-resolved rule is troublesome in the 
> case of blocks layout recalls, consider someone with an outstanding 
> LAYOUTGET for range A of the file.  Now a recall is received for a 
> disjoint range B of the file.  If the client had to wait to get A 
> before he could return B, you might have a serious performance issue 
> in that another client waiting to get B might be unduly delayed.  But 
> it is worse.  To get layout A, the server might be waiting for another

> client to return A and he might in turn be waiting for some other 
> layout.  It sounds like a performance nightmare and the possibility of

> deadlock is there too.

I sent a correction right after the message you're replying to here in
which I suggested that the server will reject any layoutget that might
conflict with an outstanding recall to the same client.

consider this example:

client 1 holds layout for range A of a file client 2 holds layout for
range B of a file

Client 1 and 2 simultaneously launch LAYOUTGETs for each other's ranges.
Assume these are for writing, so the server is going to do
CB_LAYOUTRECALL.

If the server launches both recalls, and delays its reply to the GETS
until the corresponding RETURNS, and the clients do not process any
RECALLS until in-flight GETs complete, then you deadlock, and both
recalls will eventually time out.

If the server returns an error (LAYOUTRECALL_IN_PROGRESS) to the blocked
GETs since they are considered conflicting with the outstanding RECALLs
then both clients can promptly return their ranges.
If the server does that only when it detects a deadlock (a loop in a
chain of GETs and RECALLs) I think that the performance penalty is
tolerable (i.e. waiting for a open ended chain of GETs and RECALLs to
unwind via successful returns of all GETs in the chain), otherwise if
the server always rejects GETs as I said clients progress will slow down
when there's heavy layout GET/RECALL activity from the same client on a
file.

> 
> So here is where I think sessions (with the multiple slotid/sequence 
> pairs but without SLOTID_I_DONT_FEEL_LIKE_TELLING_YOU :-) can deal 
> with the issue.  The issue is that if the client is not told by the 
> server what LAYOUTGET's he has to wait for, he has to wait for all of 
> the LAYOUTGET's that might conflict and as you point out, that can be 
> a larger set than you'd like.  The point here is that the client only 
> knows what he asked for while the server knows what the client was 
> given (including zero in the case of LAYOUTGET's he hasn't received).

> So in the case of LAYOUTRETURN prompted by a recall the client has a 
> way to avoid superfluously waiting for LAYOUTGET's that have only 
> potential conflicts.  Note here that a LAYOUTRETURN prompted by a 
> recall is precisely the case in which the return has a strong need for

> a prompt return to satisfy the recall.

If I understand your point correctly you say that if the client knows
that an outstanding LAYOUTGET was seen by the server it doesn't need to
wait for the reply before returning layout.

let's see if this helps resolving the deadline scenario discussed
earlier.

client 1 holds layout for range A of a file client 2 holds layout for
range B of a file

client 1 sends GET for range B, and in parallel client 2 sends GET for
range A server receives GET for range A from client 2

server sends RECALL for range A to client 1

case 1: server haven't seen the GET for range B from client 1 client 1
waits for its GET for range B to complete

server receives this GET, recalls range B from client 2 client 2 is told
its GET for range A was seen so it doesn't wait for it and returns
layout for range B does this complete its LAYOUTRECALL?

client 1 gets its layout for range B
returns layout for range A, recall complete

client 2 gets its layout for range A, recall complete

OK, now say the server sent whole file recalls for whatever reason the
clients can't return the whole range until they receive replies for
their outstanding GETs (Whether seen or not by the server).
When they do get them they need to send the final RETURN for the whole
file.  As I said, the server can expedite the process by rejecting the
outstanding GETs since they conflict with the RECALL it sent.

I think that works. Just remember that the client still needs to wait
for all outstanding layoutgets to complete before completing processing
of the layoutrecall since it might need to eventually return the layout
it will get in the future if it conflicts with the recall it is
processing now, or, from another perspective, if it conflicts with any
of the returns that it sent when the outstanding GET was in flight.

This applies to voluntary returns as well.
I still think that the simplest approach for client implementations is
to serialize LAYOUTRETURN with any outstanding LAYOUTGET on the same
file (or conflicting with it when we have LAYOUTRETURNs for FSID/ALL)
otherwise, the client will need to track for each outstanding LAYOUTGET
all possibly conflicting LAYOUTRETURNs sent while it was in flight
(whether voluntary or not, it doesn't matter), and when the LAYOUTGET
returns check the retrieved layout parameters against the said RETURNs
and return the layout (or parts of it) if any of the RETURNs conflict
with it.

> 
> 
>>My interpretation is that there's a hidden assumption that there's a
> 
> recall
> 
>>meta-operation that starts when a CB_LAYOUTRECALL is issued and is
> 
> complete when
> 
>>all layout segments covered by the layout recall scope are returned.
> 
> Since we
> 
>>don't mandate an explicit return operation that matches the respective
> 
> recall
> 
>>operation there's a potential for the client and server to go or
> 
> remain out of
> 
>>sync here.
> 
> 
> I believe that is the assumption.  I don't see that assumption is a 
> problem as long as the client and server remain in sync with regard to

> layout state.  If they are in sync, then determining when all segments

> have been returned is fine.  I think once we allow the client and 
> server to get out of sync, we are in a bad state, anyway.
> I don't think the problem is fundamentally one of conflict between 
> LAYOUTGET and RECALL but rather of proper synchronization of LAYOUTGET

> and LAYOUTRETURN.

true.

> 
> 
>>1) The server sends CB_LAYOUTRECALL for the layouts it wants back
>>    (RECALL_FILE, RECALL_FSID, or RECALL_ALL).
> 
> 
> I don't see the need for the meta-operation.  The recall should be 
> satisfied when everything that was requested to be recalled is 
> returned and the information in CB_SEQUENCE unambiguously defines the 
> server's view of the state and which LAYOUTGETs the client has to wait

> to get resolved before doing returns.

The proposed meta-operation is not a real one, it's considered only on
the client side and used for explanatory reasons.  I don't think the
server needs to track its progress but rather rely only on the GETs and
RETURNs it processes.

> 
> 
>>2) While the recall meta-operation is in progress the client MUST NOT
> 
> issue
> 
>>   any possibly conflicting LAYOUTGET operation.
> 
> 
> I don't see the problem with him requesting such a layout.  The server

> is not going to give it to him until he thinks it is available.

but if it does grant him new layouts the final response may return them.
The client needs to be aware of that. My proposal is aimed at
simplifying the client's model. I agree it is not essential for
correctness.

> 
> 
>>3) The client MUST wait for any outstanding LAYOUTGET that possibly
> 
> conflicts
> 
>>  with the recalled layout to complete before returning any layout.
>>
>>  3.1) Wait time can be bound to the server lease time as the layouts
> 
> will be
> 
>>       revoked by the server if CB_LAYOUTRECALL is not replied by
> 
> then.
> 
> I think this is the biggest problem: excessive wait time and possible 
> deadlocks, especially for blocks.  The set of possible conflicts can 
> be too large.
> With
> sessions, you only have to wait for actual conflicts on a recall and 
> that makes things workable.
> 
> Note that this means that 3) means I can't return the layout for file 
> A while I'm waiting for LAYOUTGET's to complete on file B, even if I 
> don't have any outstanding LAYOUTGET's for file B.  That make 
> recall-fsid very deadlock prone.
> 
> 
>>4) Possible conflicts are determined by matching clientid,
> 
> layout_type, iomode,
> 
>>   and depending on the recall scope (RECALL_{FILE,FSID,ALL})
> 
> outstanding
> 
>>   LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly
> 
> conflicting.
> 
>>   Note that since the server can return a layout segment which is
> 
> outside
> 
>>   the requested byte range in LAYOUTGET, it may conflict with the
>>   CB_LAYOUTRECALL even if the recalled byte range does not overlap
> 
> with the
> 
>>   byte range requested in LAYOUTGET, hence the conflict is assumed on
> 
> any
> 
>>   LAYOUTGET to the same file, with a conflicting layout_type, iomode
> 
> and
> 
>>   clientid.
> 
> 
> This is workable for voluntary returns but problematic for recalls for

> reasons already discussed.
> 
> 
>>5) At this point, if the client finds no matching layout it returns a
>>    NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.
> 
> 
> I don't see that it is necessary to give the server an error if you 
> did a voluntary return of something and he recalled it before he got 
> the return.
> There is no divergence of state here.  Both client and server agree 
> except for the unavoidable difference because A hasn't got B's request

> yet.

If there's an in-flight return then the client can't assume the layout
was returned until the RETURN is replied since it can fail.
If there's no return in flight and the client knows of no layout it
needs to return it can just send the final return to "confirm"
the recall.  It can server the same purpose of this error.
Therefore, I have no problem with removing the error from the protocol
and rely solely only LAYOUTGETs and RETURNs to maintain the layout
state.

> 
> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]]
> Sent: Thursday, June 22, 2006 12:55 PM
> To: [email-masked]
> Subject: [nfsv4] pnfs: keeping the client and server layouts in sync
> 
> 
> While working on Issue 58 I became quite concerned about keeping the 
> client and server layout state in sync with the current layout 
> get/recall/return protocol when there are concurrent conflicting get 
> and return or recall calls in flight even if we use sessions to order 
> the calls.  My concern arose from the fuzziness of LAYOUTRETURN that 
> allows "a subset or superset of a layout specified by CB_LAYOUTRECALL"

> and "It is also permissible, and no error should result, for a client 
> to return a byte range covering a layout it does not hold".
> 
> When a layoutget is in flight the state of holding a layout segment 
> transfers from the server to the client and until the client receives 
> the response for the layoutget it doesn't really know if it already 
> holds the layout or not.
> Moreover, since the server can return a layout segment outside the 
> segment the client asked for, even if the outstanding requested range 
> and the about-to-be-sent layoutreturn do not intersect, there might be

> a potential conflict on the "extended" range that the server is about 
> to give to the client.
>   For example:
> 1. client holds a layout for offset 0, length 3 2. client returns 
> offset 0, length 2 3. client sends a layoutget for offset 0, length 1 
> 4. client sends a layoutreturn for offset 1, length 2 5. server 
> replies (3) for offset 0, length 2
> 
> note the conflict on the byte at offset 1.  At stages 3 and 4 the 
> client does not know it is going to be the proud owner of the layout 
> segment at byte
> 1 since
> it didn't get (5) in time. When (4) is received the server will think 
> that the client holds a layout for <offset=0, length=1> and the client

> will think, after receiving (5) that it holds a layout for <offset=1, 
> length=2>. Oops.
> 
> This becomes more obvious if the client does a return for RETURN_FSID 
> (as proposed in Issue 58) in which conflicts are more abundant.
> Unfortunately, not
> allowing RETURN_FSID helps only to hide problems by allowing the 
> client to remain out of sync with the server without that being 
> noticed.  Also the text for "Recalling a layout" (Section 14.5.1) is 
> challenging for a CB_LAYOUTRECALL for RECALL_FSID
> 
>     In order to ensure client/server convergence on the layout state, 
> the final
>     LAYOUTRETURN operation in a sequence of returns for a particular 
> recall,
>     SHOULD specify the entire range being recalled, even if layout 
> segments
>     pertaining to partial ranges were previously returned.
> 
> This adds to LAYOUTRETURN semantics (Section 22.48)
> 
>     However, if it is a subset, the recall is not complete until the 
> full
>     byte range has been returned.
> 
> My interpretation is that there's a hidden assumption that there's a 
> recall meta-operation that starts when a CB_LAYOUTRECALL is issued and

> is complete when all layout segments covered by the layout recall 
> scope are returned.
> Since we
> don't mandate an explicit return operation that matches the respective

> recall operation there's a potential for the client and server to go 
> or remain out of sync here.
> 
> I think that to allow the server to clean up the client state in a 
> sensible way we need to require such a final response and add some 
> other restrictions on the client as described below so that the state 
> of the client's layout is safely synchronized between the client and 
> the server when the client receives the reply for the final return it 
> sent in response to a CB_LAYOUTRECALL.
> 
> 1) The server sends CB_LAYOUTRECALL for the layouts it wants back
>     (RECALL_FILE, RECALL_FSID, or RECALL_ALL).
> 
>     1.1) This triggers a recall meta-operation at the client in which 
> it will
>          return all layouts matching the CB_LAYOUTRECALL parameters.
> 
>     1.2) The meta-operation completes when either CB_LAYOUTRECALL is 
> replied with
>          an error, or when the final LAYOUTRETURN reply is received by

> the
>          client.
> 
>     1.2) If at any point during the meta-operation the server 
> restarts, then
>          regular server recovery applies and the client implicitly 
> loses all its
>          layouts.
> 
> 2) While the recall meta-operation is in progress the client MUST NOT 
> issue
>     any possibly conflicting LAYOUTGET operation.
> 
> 3) The client MUST wait for any outstanding LAYOUTGET that possibly 
> conflicts
>     with the recalled layout to complete before returning any layout.
> 
>     3.1) Wait time can be bound to the server lease time as the 
> layouts will be
>          revoked by the server if CB_LAYOUTRECALL is not replied by 
> then.
> 
> 4) Possible conflicts are determined by matching clientid, 
> layout_type, iomode,
>     and depending on the recall scope (RECALL_{FILE,FSID,ALL}) 
> outstanding
>     LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly 
> conflicting.
>     Note that since the server can return a layout segment which is 
> outside
>     the requested byte range in LAYOUTGET, it may conflict with the
>     CB_LAYOUTRECALL even if the recalled byte range does not overlap 
> with the
>     byte range requested in LAYOUTGET, hence the conflict is assumed 
> on any
>     LAYOUTGET to the same file, with a conflicting layout_type, iomode

> and
>     clientid.
> 
> 5) At this point, if the client finds no matching layout it returns a
>     NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.
> 
> 6) Otherwise, the client starts returning the matching layouts.
>     6.1) It can do this in a single or multiple LAYOUTRETURN 
> operations as long
>          as the final LAYOUTRETURN call echoes exactly the same 
> parameters of the
>          respective CB_LAYOUTRECALL.
> 
>     6.1) To make sure that the final LAYOUTRETURN is received last, 
> the client
>          SHOULD wait for any outstanding LAYOUTRETURNs to be replied 
> (if it sent
>          any) before sending the final one.  With sessions, if all 
> respective
>          LAYOUTRETURN operations are serialized on the same slot, the 
> final
>          LAYOUTRETURN will naturally happen last.
> 
> 7) The server should rely only on the LAYOUTRETURNs it receives from 
> the client
>     so it does not have to maintain any state that is associated with
a
>     CB_LAYOUTRECALL it sent to a client.
>     7.1) If the server wants to sync with the client layout state it 
> can send a
>          "wildcard" CB_LAYOUTRECALL (whole file, fsid, or all layouts)
>          and wait until the final LAYOUTRETURN is received from the 
> client. At
>          this point the server and client states (for the given 
> clientid,
>          layout_type, iomode, and scope) can be safely reset.
> 
> I believe that doing the above will save us from relying on sessions 
> to resolve recall/get races.
> 
> Benny
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Tue, 27 Jun 2006 18:06:18 +0300

> what you are worried about. 

My worries are broader then the specific issue of using sessions
for CB_LAYOUTRECALL. I'm OK with the proposed "no SLOTID_UNKNOWN"
approach for issue 19.

Benny