Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in sync

From: Benny Halevy <[email-masked]>
Date: Tue, 27 Jun 2006 11:58:46 +0300

Dave, I'll try to summarize my observations here, more discussion below...

LAYOUTGETs and RETURNs modify the layout state on both sides of the wire.

The server owns the authoritative state and the changes to the state flow
from the server to the client via respective replies to GETs and RETURNs.

To stay in sync they need to be serialized and applied in exactly the same
order on both sides of the wire.

The client can implement a simple serialization scheme by blocking any
GET or RETURN on possibly conflicting outstanding RETURNs or GETs.

A special case is CB_LAYOUTRECALL in which the client can assume
and avoid a conflict based on the recalled range rather than on any
specific LAYOUTRETURN.

Deadlock prevention can be done on the server by rejecting GETs that
are possibly conflicting with RECALLs the server sends until the
server gets the reply for that CB_LAYOUTRECALL.  Sessions can probably
be used here to fine tune this.

It might be possible to utilize sessions to serialize GETs and RETURNs
but the server will have to strictly order these operations so that the
client will know exactly which outstanding LAYOUTGETs were affected
by a RETURN it sent, and even then the client will probably have to guess
what the effect was or "force" a state sync by returning these layouts
after it got them if it's not sure how to resolve the conflict.

I think that trying to optimize potential performance issues using
sessions is going to be hard and I wouldn't want to risk correctness
while we're working on the problem. My gut feeling is that to be on the
safe side we should start with the simple model that does not rely on
sessions (and prevent deadlocks as described above) and then work out
a solution that uses sessions to optimize performance.

Benny

Noveck, Dave wrote:
> I think you raise some good points but I also disagree with your 
> analysis in part.
> 
> First of all, I agree that the current situation with LAYOUTRETURN is
> worrisome, even apart from issues specific to recall.  I think there 
> just seem to be too many ways that the client and server can get out
> of sync.  The text you quote "In order to ensure client/server
> convergence 
> on the layout state" makes me especially nervous since convergence
> would only be needed if there a previous divergence to converge from,
> and it isn't really clear how we can both allow that divergence 
> while guaranteeing later convergence.

agreed.

> 
> The important point is that the range of possible conflicts when
> you have outstanding LAYOUTGET's may be larger than you might expect
> and this has important consequence which I'll talk about later.
> 
> I don't agree that the issue becomes worse when recalling fsid's.
> It is true that there may be more conflicts in this case but all
> of the conflicts are knowable by the client he has issued the LAYOUTGET.
> Since a recall of an FSID recalls the entire range file there is
> no question of a hidden conflict where the range you get is bigger
> than what you ask for.  In this case, you should not return a layout
> while you have a conflicting LAYOUTGET in the pipe.  You simply have
> to wait to resolve the LAYOUTGET before doing the LAYOUTRETURN.  If
> you don't, the client's view and the servers view of things are likely
> to diverge.
> 
> Now in the case of a LAYOUTRETURN for a small range, we still have the
> same issue.  Any LAYOUTGET, as you note, represents a potential conflict
> even if the range sought does not overlap the range to be returned.
> So it looks like the only way that you can keep the server and client
> in sync is to say you can't do the LAYOUTRETURN when any LAYOUTGET
> for the same file is outstanding.  

This is important.

> I think this is workable in the
> case of voluntary return but poses problems in the case of recall, but
> that sessions, provides a way out of the problem.  Unlike you, I think
> you really need sessions to get around some of the recall issues.
> 
> To see why a wait-until-GET-is-resolved rule is troublesome in the
> case of blocks layout recalls, consider someone with an outstanding
> LAYOUTGET for range A of the file.  Now a recall is received for a
> disjoint range B of the file.  If the client had to wait to get 
> A before he could return B, you might have a serious performance
> issue in that another client waiting to get B might be unduly 
> delayed.  But it is worse.  To get layout A, the server might be 
> waiting for another client to return A and he might in turn be
> waiting for some other layout.  It sounds like a performance nightmare
> and the possibility of deadlock is there too.

I sent a correction right after the message you're replying to here
in which I suggested that the server will reject any layoutget that
might conflict with an outstanding recall to the same client.

consider this example:

client 1 holds layout for range A of a file
client 2 holds layout for range B of a file

Client 1 and 2 simultaneously launch LAYOUTGETs for each other's ranges.
Assume these are for writing, so the server is going to do CB_LAYOUTRECALL.

If the server launches both recalls, and delays its reply
to the GETS until the corresponding RETURNS, and the clients
do not process any RECALLS until in-flight GETs complete,
then you deadlock, and both recalls will eventually time out.

If the server returns an error (LAYOUTRECALL_IN_PROGRESS) to the
blocked GETs since they are considered conflicting with the
outstanding RECALLs then both clients can promptly return their ranges.
If the server does that only when it detects a deadlock (a loop in a
chain of GETs and RECALLs) I think that the performance penalty is
tolerable (i.e. waiting for a open ended chain of GETs and RECALLs
to unwind via successful returns of all GETs in the chain), otherwise
if the server always rejects GETs as I said clients progress will slow
down when there's heavy layout GET/RECALL activity from the same client
on a file.

> 
> So here is where I think sessions (with the multiple slotid/sequence
> pairs but without SLOTID_I_DONT_FEEL_LIKE_TELLING_YOU :-) can deal
> with the issue.  The issue is that if the client is not told by the
> server what LAYOUTGET's he has to wait for, he has to wait for all
> of the LAYOUTGET's that might conflict and as you point out, that
> can be a larger set than you'd like.  The point here is that the client
> only knows what he asked for while the server knows what the client
> was given (including zero in the case of LAYOUTGET's he hasn't
> received).  So in the case of LAYOUTRETURN prompted by a recall
> the client has a way to avoid superfluously waiting for LAYOUTGET's
> that have only potential conflicts.  Note here that a LAYOUTRETURN
> prompted by a recall is precisely the case in which the return has
> a strong need for a prompt return to satisfy the recall.

If I understand your point correctly you say that if the client knows
that an outstanding LAYOUTGET was seen by the server it doesn't
need to wait for the reply before returning layout.

let's see if this helps resolving the deadline scenario discussed earlier.

client 1 holds layout for range A of a file
client 2 holds layout for range B of a file

client 1 sends GET for range B, and in parallel
client 2 sends GET for range A
server receives GET for range A from client 2

server sends RECALL for range A to client 1

case 1: server haven't seen the GET for range B from client 1
client 1 waits for its GET for range B to complete

server receives this GET, recalls range B from client 2
client 2 is told its GET for range A was seen so
it doesn't wait for it and returns layout for range B
does this complete its LAYOUTRECALL?

client 1 gets its layout for range B
returns layout for range A, recall complete

client 2 gets its layout for range A, recall complete

OK, now say the server sent whole file recalls for whatever reason
the clients can't return the whole range until they receive replies
for their outstanding GETs (Whether seen or not by the server).
When they do get them they need to send the final RETURN for the
whole file.  As I said, the server can expedite the process by
rejecting the outstanding GETs since they conflict with the RECALL
it sent.

I think that works. Just remember that the client still needs
to wait for all outstanding layoutgets to complete before
completing processing of the layoutrecall since it might
need to eventually return the layout it will get in the future
if it conflicts with the recall it is processing now, or,
from another perspective, if it conflicts with any of the returns
that it sent when the outstanding GET was in flight.

This applies to voluntary returns as well.
I still think that the simplest approach for client implementations
is to serialize LAYOUTRETURN with any outstanding LAYOUTGET on the
same file (or conflicting with it when we have LAYOUTRETURNs for FSID/ALL)
otherwise, the client will need to track for each outstanding LAYOUTGET
all possibly conflicting LAYOUTRETURNs sent while it was in flight
(whether voluntary or not, it doesn't matter), and when the LAYOUTGET
returns check the retrieved layout parameters against the said RETURNs
and return the layout (or parts of it) if any of the RETURNs
conflict with it.

> 
> 
>>My interpretation is that there's a hidden assumption that there's a
> 
> recall
> 
>>meta-operation that starts when a CB_LAYOUTRECALL is issued and is
> 
> complete when
> 
>>all layout segments covered by the layout recall scope are returned.
> 
> Since we
> 
>>don't mandate an explicit return operation that matches the respective
> 
> recall
> 
>>operation there's a potential for the client and server to go or
> 
> remain out of
> 
>>sync here.
> 
> 
> I believe that is the assumption.  I don't see that assumption is a
> problem as long as the client and server remain in sync with regard
> to layout state.  If they are in sync, then determining when all 
> segments have been returned is fine.  I think once we allow the 
> client and server to get out of sync, we are in a bad state, anyway.
> I don't think the problem is fundamentally one of conflict between 
> LAYOUTGET and RECALL but rather of proper synchronization of LAYOUTGET 
> and LAYOUTRETURN.

true.

> 
> 
>>1) The server sends CB_LAYOUTRECALL for the layouts it wants back
>>    (RECALL_FILE, RECALL_FSID, or RECALL_ALL).
> 
> 
> I don't see the need for the meta-operation.  The recall should be
> satisfied
> when everything that was requested to be recalled is returned and the 
> information in CB_SEQUENCE unambiguously defines the server's view of
> the state and which LAYOUTGETs the client has to wait to get resolved 
> before doing returns.

The proposed meta-operation is not a real one, it's considered only
on the client side and used for explanatory reasons.  I don't think
the server needs to track its progress but rather rely only on the GETs
and RETURNs it processes.

> 
> 
>>2) While the recall meta-operation is in progress the client MUST NOT
> 
> issue
> 
>>   any possibly conflicting LAYOUTGET operation.
> 
> 
> I don't see the problem with him requesting such a layout.  The server
> is
> not going to give it to him until he thinks it is available. 

but if it does grant him new layouts the final response may return them.
The client needs to be aware of that. My proposal is aimed at simplifying
the client's model. I agree it is not essential for correctness.

> 
> 
>>3) The client MUST wait for any outstanding LAYOUTGET that possibly
> 
> conflicts
> 
>>  with the recalled layout to complete before returning any layout.
>>
>>  3.1) Wait time can be bound to the server lease time as the layouts
> 
> will be
> 
>>       revoked by the server if CB_LAYOUTRECALL is not replied by
> 
> then.
> 
> I think this is the biggest problem: excessive wait time and possible
> deadlocks,
> especially for blocks.  The set of possible conflicts can be too large.
> With
> sessions, you only have to wait for actual conflicts on a recall and
> that makes
> things workable.
> 
> Note that this means that 3) means I can't return the layout for file A
> while I'm waiting for LAYOUTGET's to complete on file B, even if I don't
> have any outstanding LAYOUTGET's for file B.  That make recall-fsid
> very deadlock prone.
> 
> 
>>4) Possible conflicts are determined by matching clientid,
> 
> layout_type, iomode,
> 
>>   and depending on the recall scope (RECALL_{FILE,FSID,ALL})
> 
> outstanding
> 
>>   LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly
> 
> conflicting.
> 
>>   Note that since the server can return a layout segment which is
> 
> outside
> 
>>   the requested byte range in LAYOUTGET, it may conflict with the
>>   CB_LAYOUTRECALL even if the recalled byte range does not overlap
> 
> with the
> 
>>   byte range requested in LAYOUTGET, hence the conflict is assumed on
> 
> any
> 
>>   LAYOUTGET to the same file, with a conflicting layout_type, iomode
> 
> and
> 
>>   clientid.
> 
> 
> This is workable for voluntary returns but problematic for recalls for
> reasons 
> already discussed.
> 
> 
>>5) At this point, if the client finds no matching layout it returns a
>>    NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.
> 
> 
> I don't see that it is necessary to give the server an error if you did
> a voluntary return of something and he recalled it before he got the
> return.
> There is no divergence of state here.  Both client and server agree
> except
> for the unavoidable difference because A hasn't got B's request yet.

If there's an in-flight return then the client can't assume the layout
was returned until the RETURN is replied since it can fail.
If there's no return in flight and the client knows of no layout
it needs to return it can just send the final return to "confirm"
the recall.  It can server the same purpose of this error.
Therefore, I have no problem with removing the error from the protocol
and rely solely only LAYOUTGETs and RETURNs to maintain the layout
state.

> 
> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Thursday, June 22, 2006 12:55 PM
> To: [email-masked]
> Subject: [nfsv4] pnfs: keeping the client and server layouts in sync
> 
> 
> While working on Issue 58 I became quite concerned about keeping the
> client and
> server layout state in sync with the current layout get/recall/return
> protocol
> when there are concurrent conflicting get and return or recall calls in
> flight
> even if we use sessions to order the calls.  My concern arose from the
> fuzziness
> of LAYOUTRETURN that allows "a subset or superset of a layout specified
> by
> CB_LAYOUTRECALL" and "It is also permissible, and no error should
> result, for a
> client to return a byte range covering a layout it does not hold".
> 
> When a layoutget is in flight the state of holding a layout segment
> transfers
> from the server to the client and until the client receives the response
> for the
> layoutget it doesn't really know if it already holds the layout or not.
> Moreover, since the server can return a layout segment outside the
> segment the
> client asked for, even if the outstanding requested range and the
> about-to-be-sent layoutreturn do not intersect, there might be a
> potential
> conflict on the "extended" range that the server is about to give to the
> client.
>   For example:
> 1. client holds a layout for offset 0, length 3
> 2. client returns offset 0, length 2
> 3. client sends a layoutget for offset 0, length 1
> 4. client sends a layoutreturn for offset 1, length 2
> 5. server replies (3) for offset 0, length 2
> 
> note the conflict on the byte at offset 1.  At stages 3 and 4 the client
> does
> not know it is going to be the proud owner of the layout segment at byte
> 1 since
> it didn't get (5) in time. When (4) is received the server will think
> that the
> client holds a layout for <offset=0, length=1> and the client will
> think, after
> receiving (5) that it holds a layout for <offset=1, length=2>. Oops.
> 
> This becomes more obvious if the client does a return for RETURN_FSID
> (as
> proposed in Issue 58) in which conflicts are more abundant.
> Unfortunately, not
> allowing RETURN_FSID helps only to hide problems by allowing the client
> to
> remain out of sync with the server without that being noticed.  Also the
> text
> for "Recalling a layout" (Section 14.5.1) is challenging for a
> CB_LAYOUTRECALL
> for RECALL_FSID
> 
>     In order to ensure client/server convergence on the layout state,
> the final
>     LAYOUTRETURN operation in a sequence of returns for a particular
> recall,
>     SHOULD specify the entire range being recalled, even if layout
> segments
>     pertaining to partial ranges were previously returned.
> 
> This adds to LAYOUTRETURN semantics (Section 22.48)
> 
>     However, if it is a subset, the recall is not complete until the
> full
>     byte range has been returned.
> 
> My interpretation is that there's a hidden assumption that there's a
> recall
> meta-operation that starts when a CB_LAYOUTRECALL is issued and is
> complete when
> all layout segments covered by the layout recall scope are returned.
> Since we
> don't mandate an explicit return operation that matches the respective
> recall
> operation there's a potential for the client and server to go or remain
> out of
> sync here.
> 
> I think that to allow the server to clean up the client state in a
> sensible way
> we need to require such a final response and add some other restrictions
> on the
> client as described below so that the state of the client's layout is
> safely
> synchronized between the client and the server when the client receives
> the
> reply for the final return it sent in response to a CB_LAYOUTRECALL.
> 
> 1) The server sends CB_LAYOUTRECALL for the layouts it wants back
>     (RECALL_FILE, RECALL_FSID, or RECALL_ALL).
> 
>     1.1) This triggers a recall meta-operation at the client in which it
> will
>          return all layouts matching the CB_LAYOUTRECALL parameters.
> 
>     1.2) The meta-operation completes when either CB_LAYOUTRECALL is
> replied with
>          an error, or when the final LAYOUTRETURN reply is received by
> the
>          client.
> 
>     1.2) If at any point during the meta-operation the server restarts,
> then
>          regular server recovery applies and the client implicitly loses
> all its
>          layouts.
> 
> 2) While the recall meta-operation is in progress the client MUST NOT
> issue
>     any possibly conflicting LAYOUTGET operation.
> 
> 3) The client MUST wait for any outstanding LAYOUTGET that possibly
> conflicts
>     with the recalled layout to complete before returning any layout.
> 
>     3.1) Wait time can be bound to the server lease time as the layouts
> will be
>          revoked by the server if CB_LAYOUTRECALL is not replied by
> then.
> 
> 4) Possible conflicts are determined by matching clientid, layout_type,
> iomode,
>     and depending on the recall scope (RECALL_{FILE,FSID,ALL})
> outstanding
>     LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly
> conflicting.
>     Note that since the server can return a layout segment which is
> outside
>     the requested byte range in LAYOUTGET, it may conflict with the
>     CB_LAYOUTRECALL even if the recalled byte range does not overlap
> with the
>     byte range requested in LAYOUTGET, hence the conflict is assumed on
> any
>     LAYOUTGET to the same file, with a conflicting layout_type, iomode
> and
>     clientid.
> 
> 5) At this point, if the client finds no matching layout it returns a
>     NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.
> 
> 6) Otherwise, the client starts returning the matching layouts.
>     6.1) It can do this in a single or multiple LAYOUTRETURN operations
> as long
>          as the final LAYOUTRETURN call echoes exactly the same
> parameters of the
>          respective CB_LAYOUTRECALL.
> 
>     6.1) To make sure that the final LAYOUTRETURN is received last, the
> client
>          SHOULD wait for any outstanding LAYOUTRETURNs to be replied (if
> it sent
>          any) before sending the final one.  With sessions, if all
> respective
>          LAYOUTRETURN operations are serialized on the same slot, the
> final
>          LAYOUTRETURN will naturally happen last.
> 
> 7) The server should rely only on the LAYOUTRETURNs it receives from the
> client
>     so it does not have to maintain any state that is associated with a
>     CB_LAYOUTRECALL it sent to a client.
>     7.1) If the server wants to sync with the client layout state it can
> send a
>          "wildcard" CB_LAYOUTRECALL (whole file, fsid, or all layouts)
>          and wait until the final LAYOUTRETURN is received from the
> client. At
>          this point the server and client states (for the given
> clientid,
>          layout_type, iomode, and scope) can be safely reset.
> 
> I believe that doing the above will save us from relying on sessions to
> resolve 
> recall/get races.
> 
> Benny
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Tue, 27 Jun 2006 07:15:53 -0400

>a solution that uses sessions to optimize performance.

We've done that, and unfortunately it leads to a dead end on correctness.
You end up with the possibility of a blocked v4.0 server waiting for a client
which is trying desperately to reconnect its operations channel. This forces
the server to revoke the layout, or the client to return NFS4ERR_DELAY,
which is nearlythe same thing.

The session names each operation, which allows the server to tell the client
exactly what was outstanding at the time of the recall. Additionally, the session
allows the client to use the same connection for operations and callbacks,
which eliminates the callback-operations 2-channel issue. While a session
doesn't close the race, it does allow the client to unequivocally determine
that it has happened, and recover appropriately.

I agree with everything else you said, but not that sessions are merely a
performance optimization. They are for correctness.

Tom.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 27 Jun 2006 22:28:49 +0300

> performance optimization. They are for correctness.

If the server is going to reject outstanding LAYOUTGETs that conflict
with a CB_LAYOUTRECALL it sent to the same client (similar to
what is suggested in section 14.5.3.1 for deadlock avoidance in
the absence of sessions) then I don't think sessions are
absolutely required for CB_LAYOUTRECALL, or am I missing something?
Can you provide a concrete example where we fall short without
sessions?

Note the paragraph in 14.5.3.2 about GET/RETURN races:

    This leads to a more general problem; in the absence of a callback if
    a client issues concurrent overlapping LAYOUTGET and LAYOUTRETURN
    operations, it is possible for the server to process them in either
    order.  Again, a client must take the appropriate precautions in
    serializing its actions.

Using sessions in CB_LAYOUTRECALL allows the client to serialize a bit
more efficiently as it doesn't have to wait on seen LAYOUTGETs before
issuing LAYOUTRETURNs, but I still think this complicates the client
since once it does that it has potentially conflicting concurrent
GETs and RETURNs in flight so it will have to wait on both to complete
and discard any layout it got that is overlapping with a RETURN it sent
when the GET was in flight.

Benny

---

From: Brent Welch <[email-masked]>
Date: Tue, 27 Jun 2006 15:02:41 -0700

This is a long, interesting thread.  Unfortunately for you readers,
I seem to have generated a long addition to it.  The executive summary
is:
  I talk about ordering and deadlock.
  I talk about a state generation model to reason about the problem.
  I discuss if using session sequence information is a good way
    to represent the state generation model.
  I conclude that using session sequence information can be correct,
    but I would prefer a cleaner protocol that makes the state
    generation explicit.

Issue 1.  Are the client and server in sync, or do they get confused
by the inherit races caused by callbacks.  Note that we have further
complicated things because LAYOUTGET can return a wider/different range
than requested, CB_LAYOUTRECALL can recall a narrower/different range
than returned from LAYOUTGET, and LAYOUTRETURN can return yet a different
range.

Issue 2.  Deadlocks caused by anything in the protocol that causes one
party to wait for something before completing something else.

These issues are related because the problems exposed by Issue 1
can lead you to want to simplify things.  This leads to ordering
constraints that bring us to Issue 2.

To give you a flavor for the issue, suppose:
Client 1 holds layout A that goes from 0 to 1M on some file.
Client 2 holds layout B that goes from 1M+1 to 2M on the same file.

Simultaneously, these two clients issue LAYOUTGET for the range
held by the other.  Assume these are for writing, so the server's
job is to ensure only one client holds a layout for a given range.

The server might then issue two concurrent CB_LAYOUTRECALL to the
two clients, before replying to their LAYOUTGET.

No problem, you say, because the ranges don't overlap, the client
could LAYOUTRETURN what they hold, (after responding to CB_LAYOUTRECALL),
and wait for the outstanding LAYOUTGET's to return.

But, suppose the server does something slightly unexpected.  It
decides that Client 1 deserves a layout from 0 to 2MB, and that
Client 2 is out of luck.  What in the protocol helps prevent this,
and helps the client and server avoid problems caused by this
behavior?  There is nothing in the protocol that says, "it would
be silly for the server to respond to a LAYOUTGET with a range
that overlaps with an outstanding CB_LAYOUTRECALL", or even more
realistic, there isn't anything in the protocol to prevent a
client from piggy-backing a LAYOUTGET onto the LAYOUTRETURN to
try and get the layout back immediately.

I think there are two general ways to deal with these issues.
The first is a "logical clock" or "state sequence" view.
The second is a "do one thing at a time" view.

If you do one thing at a time (e.g., HighRoad serializes
callbacks) then you need to detect conflicting operations and
reject some of them.  In our experience, you want the server
to reject things, not the client.   We have NFS4ERR_RETRY that
the server could return to LAYOUTGET, for example, if it is
in the middle of doing recalls and prefers the do one thing
at a time model.  If a server has a lock on the data structure
that represents a file, it is fairly natural for an implementation
to want to keep that lock held.  But, if you do that, you MUST
detect and reject other client operations would lead to deadlock.

The state sequence view lets operations run concurrently, and
uses some sequencing information to put things back into the
right order when there are races.  It can make implementations
more complex, especially if you back into it and don't have
a first class way to reason about ordering.  I'm a little worried
that using session sequence numbers is like this.  However,
I think it can hold water.  But, I'll explain the state
sequence model first.

Suppose the server had a layout state generation number
(ok, some would call that a stateid) that incremented:
on each LAYOUTGET reply
on each LAYOUTRETURN reply
and was transmitted to the client on each CB_LAYOUTRECALL.
In this model the server owns that generation number, and
there is just one no matter how many clients there are.
The server does not need to keep a lock held on the file
while the RECALL is in-flight.  It only keeps a lock to
serialize updates to the state generation number and
generation of the replies or recall requests.

The function of the generation number is so the client can
order the layout operations.  Suppose it did a LAYOUTGET
that returned (0,1MB) at time 1.  Now suppose two things
happen concurrently, another LAYOUTGET for (1MB,2MB)
and a CB_LAYOUTRECALL for (0,1MB).  The server will order
those two, one way or the other, and transmit them to the client
that can put them back in order and process them.  Going
back to my contrived example, the server might decide
that the CB_LAYOUTRECALL for (0,1MB) occured at time 2,
and then it replied to the LAYOUTGET for (1MB,2MB) at
time 3, but passed a wider range for (0,2MB).  Even though
it appears goofy that the server gave the client a layout
for a range it just recalled, the sequence number makes
it crystal clear what the client should do.  The client owns
the layout from (0,2MB).

As an aside, the client can pass the server its last view
of the state sequence number, but the server doesn't depend
on that for correctness.  It might be useful to defer
hungry clients that immediately do LAYOUTGET after LAYOUTRETURN.
(Or heck, concurrently.  With multiple slots the client could
return a layout with its left hand while asking for it back
with its right hand.)

Now, how does the state sequence model compare to the use
of session information?  For the session information, the server
returns the sequence information for all active slots, or
channels, between the client and server in the CB_SEQUENCE message
that proceeds the CB_LAYOUTRECALL. The client has to hold onto
this hunk of sequence information until those slots go idle so
you know what to do with the in-flight responses.

Does the session information handle the case of a concurrent
LAYOUTGET and LAYOUTRETURN from the same client for overlapping
regions?  The client could issue those over parallel slots and
process their replies in either order.  I think it remains
ambiguous what the state is in this case.  You need to add a
restriction that LAYOUTGET and LAYOUTRETURN are serialized 
(only if they could conflict, but the client can't tell ahead of time).
In contrast, the state generation number is a crisp manifestation of some
serialization that happened on the server, and you don't
need to serialize on the client.  Not a big deal, but we'll
need to account for this.

The state generation number would be a new attribute for files,
similar but different than the change number.  Or, perhaps it
is more like a lock stateid.  I have the impression that these are
viewed as too heavy-weight or complicated for use in here.
However, I believe there already is a logical time for the
layout state whether you make it concrete or not.  Is the
session state the best representation of it?  I'm not sure
that it is the simplest way to represent it.

I think sessions are great, by the way.  Their primary purpose is
to implicitly acknowledge server replies and therefore bound the
amount of memory in the server's reply cache.   I'm worried that
we are blurring the boundaries between the file protocol and
the transport protocol, and that the file protocol would be
cleaner if the layout state generation were represented explicitly.
The implementations will be cleaner, too.

[I don't have any further words in-lined below here]
	Brent

>>>Benny Halevy said:
 > > You end up with the possibility of a blocked v4.0 server waiting for a 
cli
     ent
 > > which is trying desperately to reconnect its operations channel. This 
forc
     es
 > > The session names each operation, which allows the server to tell the 
clie
     nt
 > > exactly what was outstanding at the time of the recall. Additionally, 
the
     session
 > > doesn't close the race, it does allow the client to unequivocally 
determin
     e
 > >>Assume these are for writing, so the server is going to do 
CB_LAYOUTRECALL
     .
 > >>>operation there's a potential for the client and server to go or 
remain
 > >>>4) Possible conflicts are determined by matching clientid, 
layout_type,

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Benny Halevy <[email-masked]>
Date: Wed, 28 Jun 2006 20:56:00 +0300

Brent Welch wrote:
> I seem to have generated a long addition to it.  
Sorry, I have a pretty long reply below :-/
> The executive summary
Brent, given no sequence numbers the server doesn't know if the client 
sent multiple request
in parallel or in sequence, for example, the server received a GET from 
a client, it replies to
it and then it receives a RETURN. How would it know whether the return 
was sent before
the client got its reply to the previous GET or after?
> there is just one no matter how many clients there are.
If we have such stateid I strongly recommend maintaining a layoutstateid per
{client, file} pair  and exchange it at least on GET and RETURN requests 
and replies
(sounds familiar?) this way the client could send any number of GETs and 
RETURNs
concurrently and the server could serialize them in an arbitrary order, 
incrementing the
layoutstateid every time the layout state changes for this client and 
report back to the
client the changes to this state via the respective GET and RETURN replies.
The client can put the replies back in order on its side and apply them 
serially in this order
on its own view of the state.

If you have one layoutstateid per file then it could be incremented by 
multiple clients'
actions and any client would see "holes" in the sequences it gets so to 
put them in
order one would have to wait for all outstanding requests before knowing 
you received
the reply out of order.  With one layoutstateid per {client, file} the 
client knows in advance
what stateid to expect (current stateid it knows about plus one) and if 
it gets a higher id
it knows it should wait for pending replies, if it gets a stateid from 
the past it knows something
really bad happened :)

Sending the current stateid on requests helps the server detect 
parallelism if it wants to
enforce serialization, e.g. regarding the example above, the server sent 
a reply with
layoutstateid X and it could reject any layoutstate-changing request 
with layoutstateid < X.
If the error would be something like NFS4ERR_BAD_LAYOUTSTATEID the 
client would know
that the server enforces serialization and adapt to the server's 
expectations.

Regarding sending a layoutstateid on CB_LAYOUTRECALL , the client can
limit tis wait until it's synchronized with the server at this 
layoutstateid before
sending RETURNs but then it still need to serialize them with outstanding
GETs no matter what.  It might be possible to think of a method for
the client to return only layouts obtained before the recall was sent
but I tend to think its complicated and not very useful. My preference would
be to leave CB_LAYOUTRECALL stateless and have the layoutstateids
only on GETs and RETURNs.
> generation of the replies or recall requests.
True, but if it wants to it can allow only RETURNs to the range its 
recalling and
allow or reject GETs from any client at its own discretion (a few 
examples, allow
GETs from one client and reject GETs from any other client, or allow 
GETs from
any client except the client being recalled, or reject all GETs because 
you want to
do something private with the file)
> [I don't have any further words in-lined below here]
[snip]