Subject: RE: [nfsv4] pnfs: keeping the client and server layouts in sync

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 22 Jun 2006 17:10:59 -0400

I think you raise some good points but I also disagree with your 
analysis in part.

First of all, I agree that the current situation with LAYOUTRETURN is
worrisome, even apart from issues specific to recall.  I think there 
just seem to be too many ways that the client and server can get out
of sync.  The text you quote "In order to ensure client/server
convergence 
on the layout state" makes me especially nervous since convergence
would only be needed if there a previous divergence to converge from,
and it isn't really clear how we can both allow that divergence 
while guaranteeing later convergence.

The important point is that the range of possible conflicts when
you have outstanding LAYOUTGET's may be larger than you might expect
and this has important consequence which I'll talk about later.

I don't agree that the issue becomes worse when recalling fsid's.
It is true that there may be more conflicts in this case but all
of the conflicts are knowable by the client he has issued the LAYOUTGET.
Since a recall of an FSID recalls the entire range file there is
no question of a hidden conflict where the range you get is bigger
than what you ask for.  In this case, you should not return a layout
while you have a conflicting LAYOUTGET in the pipe.  You simply have
to wait to resolve the LAYOUTGET before doing the LAYOUTRETURN.  If
you don't, the client's view and the servers view of things are likely
to diverge.

Now in the case of a LAYOUTRETURN for a small range, we still have the
same issue.  Any LAYOUTGET, as you note, represents a potential conflict
even if the range sought does not overlap the range to be returned.
So it looks like the only way that you can keep the server and client
in sync is to say you can't do the LAYOUTRETURN when any LAYOUTGET
for the same file is outstanding.  I think this is workable in the
case of voluntary return but poses problems in the case of recall, but
that sessions, provides a way out of the problem.  Unlike you, I think
you really need sessions to get around some of the recall issues.

To see why a wait-until-GET-is-resolved rule is troublesome in the
case of blocks layout recalls, consider someone with an outstanding
LAYOUTGET for range A of the file.  Now a recall is received for a
disjoint range B of the file.  If the client had to wait to get 
A before he could return B, you might have a serious performance
issue in that another client waiting to get B might be unduly 
delayed.  But it is worse.  To get layout A, the server might be 
waiting for another client to return A and he might in turn be
waiting for some other layout.  It sounds like a performance nightmare
and the possibility of deadlock is there too.

So here is where I think sessions (with the multiple slotid/sequence
pairs but without SLOTID_I_DONT_FEEL_LIKE_TELLING_YOU :-) can deal
with the issue.  The issue is that if the client is not told by the
server what LAYOUTGET's he has to wait for, he has to wait for all
of the LAYOUTGET's that might conflict and as you point out, that
can be a larger set than you'd like.  The point here is that the client
only knows what he asked for while the server knows what the client
was given (including zero in the case of LAYOUTGET's he hasn't
received).  So in the case of LAYOUTRETURN prompted by a recall
the client has a way to avoid superfluously waiting for LAYOUTGET's
that have only potential conflicts.  Note here that a LAYOUTRETURN
prompted by a recall is precisely the case in which the return has
a strong need for a prompt return to satisfy the recall.

> My interpretation is that there's a hidden assumption that there's a
recall
> meta-operation that starts when a CB_LAYOUTRECALL is issued and is
complete when
> all layout segments covered by the layout recall scope are returned.
Since we
> don't mandate an explicit return operation that matches the respective
recall
> operation there's a potential for the client and server to go or
remain out of
> sync here.

I believe that is the assumption.  I don't see that assumption is a
problem as long as the client and server remain in sync with regard
to layout state.  If they are in sync, then determining when all 
segments have been returned is fine.  I think once we allow the 
client and server to get out of sync, we are in a bad state, anyway.
I don't think the problem is fundamentally one of conflict between 
LAYOUTGET and RECALL but rather of proper synchronization of LAYOUTGET 
and LAYOUTRETURN.

> 1) The server sends CB_LAYOUTRECALL for the layouts it wants back
>     (RECALL_FILE, RECALL_FSID, or RECALL_ALL).

I don't see the need for the meta-operation.  The recall should be
satisfied
when everything that was requested to be recalled is returned and the 
information in CB_SEQUENCE unambiguously defines the server's view of
the state and which LAYOUTGETs the client has to wait to get resolved 
before doing returns.

> 2) While the recall meta-operation is in progress the client MUST NOT
issue
>    any possibly conflicting LAYOUTGET operation.

I don't see the problem with him requesting such a layout.  The server
is
not going to give it to him until he thinks it is available. 

> 3) The client MUST wait for any outstanding LAYOUTGET that possibly
conflicts
>   with the recalled layout to complete before returning any layout.
>
>   3.1) Wait time can be bound to the server lease time as the layouts
will be
>        revoked by the server if CB_LAYOUTRECALL is not replied by
then.

I think this is the biggest problem: excessive wait time and possible
deadlocks,
especially for blocks.  The set of possible conflicts can be too large.
With
sessions, you only have to wait for actual conflicts on a recall and
that makes
things workable.

Note that this means that 3) means I can't return the layout for file A
while I'm waiting for LAYOUTGET's to complete on file B, even if I don't
have any outstanding LAYOUTGET's for file B.  That make recall-fsid
very deadlock prone.

> 4) Possible conflicts are determined by matching clientid,
layout_type, iomode,
>    and depending on the recall scope (RECALL_{FILE,FSID,ALL})
outstanding
>    LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly
conflicting.
>    Note that since the server can return a layout segment which is
outside
>    the requested byte range in LAYOUTGET, it may conflict with the
>    CB_LAYOUTRECALL even if the recalled byte range does not overlap
with the
>    byte range requested in LAYOUTGET, hence the conflict is assumed on
any
>    LAYOUTGET to the same file, with a conflicting layout_type, iomode
and
>    clientid.

This is workable for voluntary returns but problematic for recalls for
reasons 
already discussed.

> 5) At this point, if the client finds no matching layout it returns a
>     NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.

I don't see that it is necessary to give the server an error if you did
a voluntary return of something and he recalled it before he got the
return.
There is no divergence of state here.  Both client and server agree
except
for the unavoidable difference because A hasn't got B's request yet.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, June 22, 2006 12:55 PM
To: [email-masked]
Subject: [nfsv4] pnfs: keeping the client and server layouts in sync

While working on Issue 58 I became quite concerned about keeping the
client and
server layout state in sync with the current layout get/recall/return
protocol
when there are concurrent conflicting get and return or recall calls in
flight
even if we use sessions to order the calls.  My concern arose from the
fuzziness
of LAYOUTRETURN that allows "a subset or superset of a layout specified
by
CB_LAYOUTRECALL" and "It is also permissible, and no error should
result, for a
client to return a byte range covering a layout it does not hold".

When a layoutget is in flight the state of holding a layout segment
transfers
from the server to the client and until the client receives the response
for the
layoutget it doesn't really know if it already holds the layout or not.
Moreover, since the server can return a layout segment outside the
segment the
client asked for, even if the outstanding requested range and the
about-to-be-sent layoutreturn do not intersect, there might be a
potential
conflict on the "extended" range that the server is about to give to the
client.
  For example:
1. client holds a layout for offset 0, length 3
2. client returns offset 0, length 2
3. client sends a layoutget for offset 0, length 1
4. client sends a layoutreturn for offset 1, length 2
5. server replies (3) for offset 0, length 2

note the conflict on the byte at offset 1.  At stages 3 and 4 the client
does
not know it is going to be the proud owner of the layout segment at byte
1 since
it didn't get (5) in time. When (4) is received the server will think
that the
client holds a layout for <offset=0, length=1> and the client will
think, after
receiving (5) that it holds a layout for <offset=1, length=2>. Oops.

This becomes more obvious if the client does a return for RETURN_FSID
(as
proposed in Issue 58) in which conflicts are more abundant.
Unfortunately, not
allowing RETURN_FSID helps only to hide problems by allowing the client
to
remain out of sync with the server without that being noticed.  Also the
text
for "Recalling a layout" (Section 14.5.1) is challenging for a
CB_LAYOUTRECALL
for RECALL_FSID

    In order to ensure client/server convergence on the layout state,
the final
    LAYOUTRETURN operation in a sequence of returns for a particular
recall,
    SHOULD specify the entire range being recalled, even if layout
segments
    pertaining to partial ranges were previously returned.

This adds to LAYOUTRETURN semantics (Section 22.48)

    However, if it is a subset, the recall is not complete until the
full
    byte range has been returned.

My interpretation is that there's a hidden assumption that there's a
recall
meta-operation that starts when a CB_LAYOUTRECALL is issued and is
complete when
all layout segments covered by the layout recall scope are returned.
Since we
don't mandate an explicit return operation that matches the respective
recall
operation there's a potential for the client and server to go or remain
out of
sync here.

I think that to allow the server to clean up the client state in a
sensible way
we need to require such a final response and add some other restrictions
on the
client as described below so that the state of the client's layout is
safely
synchronized between the client and the server when the client receives
the
reply for the final return it sent in response to a CB_LAYOUTRECALL.

1) The server sends CB_LAYOUTRECALL for the layouts it wants back
    (RECALL_FILE, RECALL_FSID, or RECALL_ALL).

    1.1) This triggers a recall meta-operation at the client in which it
will
         return all layouts matching the CB_LAYOUTRECALL parameters.

    1.2) The meta-operation completes when either CB_LAYOUTRECALL is
replied with
         an error, or when the final LAYOUTRETURN reply is received by
the
         client.

    1.2) If at any point during the meta-operation the server restarts,
then
         regular server recovery applies and the client implicitly loses
all its
         layouts.

2) While the recall meta-operation is in progress the client MUST NOT
issue
    any possibly conflicting LAYOUTGET operation.

3) The client MUST wait for any outstanding LAYOUTGET that possibly
conflicts
    with the recalled layout to complete before returning any layout.

    3.1) Wait time can be bound to the server lease time as the layouts
will be
         revoked by the server if CB_LAYOUTRECALL is not replied by
then.

4) Possible conflicts are determined by matching clientid, layout_type,
iomode,
    and depending on the recall scope (RECALL_{FILE,FSID,ALL})
outstanding
    LAYOUTGETs for the respective {FILE,FSID,ALL} are possibly
conflicting.
    Note that since the server can return a layout segment which is
outside
    the requested byte range in LAYOUTGET, it may conflict with the
    CB_LAYOUTRECALL even if the recalled byte range does not overlap
with the
    byte range requested in LAYOUTGET, hence the conflict is assumed on
any
    LAYOUTGET to the same file, with a conflicting layout_type, iomode
and
    clientid.

5) At this point, if the client finds no matching layout it returns a
    NFS4ERR_NOMATCHING_LAYOUT error to CB_LAYOUTRECALL.

6) Otherwise, the client starts returning the matching layouts.
    6.1) It can do this in a single or multiple LAYOUTRETURN operations
as long
         as the final LAYOUTRETURN call echoes exactly the same
parameters of the
         respective CB_LAYOUTRECALL.

    6.1) To make sure that the final LAYOUTRETURN is received last, the
client
         SHOULD wait for any outstanding LAYOUTRETURNs to be replied (if
it sent
         any) before sending the final one.  With sessions, if all
respective
         LAYOUTRETURN operations are serialized on the same slot, the
final
         LAYOUTRETURN will naturally happen last.

7) The server should rely only on the LAYOUTRETURNs it receives from the
client
    so it does not have to maintain any state that is associated with a
    CB_LAYOUTRECALL it sent to a client.
    7.1) If the server wants to sync with the client layout state it can
send a
         "wildcard" CB_LAYOUTRECALL (whole file, fsid, or all layouts)
         and wait until the final LAYOUTRETURN is received from the
client. At
         this point the server and client states (for the given
clientid,
         layout_type, iomode, and scope) can be safely reset.

I believe that doing the above will save us from relying on sessions to
resolve 
recall/get races.

Benny

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: [email-masked]
Date: Fri, 23 Jun 2006 20:51:35 -0400

Having just fought some aspects of this in issue 19 (pNFS and sessions),
I have a few thoughts to add:

> of sync.  The text you quote "In order to ensure client/server convergence

> you really need sessions to get around some of the recall issues.

I would agree.  HighRoad single-threaded all operations on a file into
a single sequence to limit this sort of opportunity for divergence.
Sessions is the corresponding NFSv4 feature to leverage.
 
recall
complete when
Since we
recall
out of
> LAYOUTGET and RECALL but rather of proper synchronization of LAYOUTGET 

Section 14.5.2 currently has a SHOULD that helps with this:

   In order to ensure client/server convergence on the layout state, the
   final LAYOUTRETURN operation in a sequence of returns for a
   particular recall, SHOULD specify the entire range being recalled,
   even if layout segments pertaining to partial ranges were previously
   returned.

I see no problem in making that a MUST.  Beyond that, I would hope that
it's not necessary to resort to passing a cookie in the recall that has
to be sent back when the return for that recall is complete.

I think Benny's discussion of meta-operation is the way that pNFS client
developers should think about recalls, even if the meta-operation is not
explicit in the protocol.  HighRoad used the reply to the recall to make
the end of the meta-operation explicit, but that's not going to work here.

satisfied
issue
> I don't see the problem with him requesting such a layout.  The server is

That's currently a "SHOULD NOT" for exactly the reason that Dave identifies
- the server MUST reject such conflicting operations if received prior to
the recall completing.  That's in 14.5.3.2, and the text changes for issue
19 make it explicit.

conflicts
will be
> >        revoked by the server if CB_LAYOUTRECALL is not replied by then.
> 
deadlocks,
With
> sessions, you only have to wait for actual conflicts on a recall and that
makes
iomode,
outstanding
conflicting.
outside
> >    CB_LAYOUTRECALL even if the recalled byte range does not overlap with
the
any
and
> reasons already discussed.
> 
return.
> There is no divergence of state here.  Both client and server agree except

I think we need a clarifying sentence here, as the last sentence of 14.5.2
isn't explicit about the case in which a return is already outstanding for
a recalled range.  As things currently stand, we'd need to clarify that
the client is considered to hold the layout until the return is
acknowledged,
although the client MUST NOT issue operations to the layout after initiating
the return operation.  Looking at the draft, we don't have good coverage
of voluntary return operations - adding a section on that just before 14.5
goes into recalls would be a good idea.

Thanks,
--David

> layout_type,

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 28 Jun 2006 17:23:14 -0400

> Problem 1 - the server has no way to enforce this serialization, 
> meaning confused
> clients can get out of sync.

Confused, broken clients will get out of sync, no matter what you
do.  That's why we call them broken.  There's nothing we can do
to prevent people who violate the spec from getting confused.  If
we implement your sequencing and I write a client that ignores
the sequence values then my client will be confused and get out of 
sync.  That doesn't there's somethng wrong with your sequencing
scheme.

> Problem 2: If there are any voluntary returns in flight to files 
> in the recalled range the client 
> needs to wait on them to return and then decide if it needs 
> to return anything else.

I don't think so.  The return of the intersection of the recalled
range and the range being returned voluntarily is adequately
protected, since the synchronization requirements for a voluntary
return are at least as strict as any recall.  Anything that is 
outside of that intersection, can be dealt with in addition, 
without waiting for the voluntary return to complete, only waiting 
for layoutgets that are designated in the CB_SEQUENCE to complete.

> Problem 3: client sent the following:
> GET #1 for range 0-1
> GET #2 for range 1-2
> server recalls range 1-2 saying he saw GET #2
> GET #2 is replied with range 0-2
> In the obj-based layout the client can't split a layout by returning 
> part of it so it has
> to return range 0-2, yet the recalled range was 1-2.
> This should fall under the voluntary return case and must be
serialized with
> GET #1. In general, if the client wants to return anything outside the
> recalled range it should be considered voluntary.

I don't see this as much of a problem.  It does seem kind of unfair for 
the server to take your get for 1-2, expand the range to 0-2 under a
rule 
where you essentially have to take the whole thing he gave you or give 
it all back, and then to top it all off he splits the range that you 
aren't allowed to split by recalling only 1-2.

> Problem 4: client works on several recalls to the same file
concurrently.
> I'm not sure what the client should do here 

I assume he works on each one independently using the rules already 
specified.  I think we can make a rule that a server not do recalls
of overalapping ranges of a file from the same client, since the
server is in a position to know what is being recalled and not 
recall what is already recalled, because there is no added value
in doing so, and because it makes things much less straightforward.-  

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Wednesday, June 28, 2006 4:42 PM
To: Noveck, Dave
Cc: Brent Welch; [email-masked]; Talpey, Thomas;
[email-masked]
Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in sync

Noveck, Dave wrote:
> As someone who went through the experience of v4.0 and has
> regretted the decision, which seemed very natural at the time,
> to add application-specific sequencing, let me insert a bit of
> a cautionary note here.  When you propose an application-specific
> sequencing mechanism, it is very easy to put together simple 
> examples that show how such a thing would work, and effectively
> ignore the problem that often makes such an approach problematic.
>
> In the steady state, there is an object for which the client and
> the server each have a sequence value and that sequence value
> provides a very helpful ordering.  Tastes great and less filling.
> The problems that are difficult turn out to be those outside that
> steady state.  What happen if an object is new to the server but
> not to the client, or vice versa.  How do you avoid having to
> keep such objects around precisely because client and server have
> a sequence value that they share and it might troublesome if one
> of the two were to drop his.  That's how we got OPEN_CONFIRM and
> RELEASE_LOCKOWNER.  These are ops that we were forced to create 
> to deal consequences of a simple per-owner sequence number.
>
> I don't know for sure that analogous problems would exist for
> this sort of sequencing but I don't think one should assume that 
> they don't exist either.  Before we go down this routine, I'd like
> to understand the motivation.
>
> What specifically is broken with the following:
>
>      Use a do-one-thing-at-a-time model for voluntary layout
>      returns, requiring LAYOUTGET's for the same file to be
>      resolved before proceeding to do the voluntary LAYOUTRETURN.
>   
That's good. This means that for each {client,file} there could be at 
any time
either GETs, RETURNs, or neither. I ignore for now RETURNs and
GETs serialized in the same compound that we may want to do
eventually...

Problem 1 - the server has no way to enforce this serialization, meaning

confused
clients can get out of sync.
>      For recall-prompted LAYOUTRETURN allow you to synchronize only
>      with LAYOUTGETs listed in the CB_SEQUENCE, with the understanding
>      that client will not grant any subsequent LAYOUTGETs (those not
>   
I assume you meant _server_ will not grant...
>      in the CB_SEQUENCE) any region that overlaps that recalled.
>   
ok, let's assume this understanding is required from the server, here 
are a few issues:

Problem 2: If there are any voluntary returns in flight to files in the 
recalled range
the client needs to wait on them to return and then decide if it needs 
to return
anything else.

Problem 3: client sent the following:
GET #1 for range 0-1
GET #2 for range 1-2
server recalls range 1-2 saying he saw GET #2
GET #2 is replied with range 0-2
In the obj-based layout the client can't split a layout by returning 
part of it so it has
to return range 0-2, yet the recalled range was 1-2.
This should fall under the voluntary return case and must be serialized
with
GET #1. In general, if the client wants to return anything outside the
recalled range it should be considered voluntary.

Problem 4: client works on several recalls to the same file
concurrently.
I'm not sure what the client should do here
> I understand that Benny has some uneasiness about the correctness
> of synchronization with this approach, but I haven't seen an example 
> presented where this approach has the client and server get out of 
> sync or where there is another significant negative consequence.
>
> Before we create another sequencing mechanism (which may wind being
> more complicated than it first appears), I'd like to be clear on the
> motivation.  Why is it that a simpler approach won't work or is 
> otherwise undesirable? 
>  
>
> From: Brent Welch [mailto:[email-masked]] 
> Sent: Wednesday, June 28, 2006 12:21 PM
> To: [email-masked]
> Cc: [email-masked]; Talpey, Thomas; [email-masked]
> Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in
sync
>
>
>
> [email-masked] said:
>   
>> I think you will still need to introduce some serialization at the
>>     
> server.
>
> The sequence number represents the serialization that has occurred.
>
> Benny points out that it is the ordering between LAYOUTGET and
> LAYOUTRETURN that is most important. The RECALL is a kind of
> hint to the client that it is supposed to do something.  (Of
> course, if you ignore the hints, the server will fence you.)
> So, let's add LAYOUTRETURN to these examples:
>
> LAYOUTGET(0,1MB) at time 1
> CB_LAYOUTRECALL(0,1MB) at time 2
> LAYOUTRETURN(0,1MB) at time X
> LAYOUTGET(1MB,2MB) at time Y, but returning (0,2MB)
>
> if (X < Y), then the server processed the return before the get,
> and the client owns (0,2MB).  This scenario is that the server
> wanted to do a recall, got the return, but then "changed its mind"
> and decided to give the layout back to the client.
>
> If (Y < X), then the server processed the get first and then
> the return, so the client owns (1MB,2MB).  This matches our
> expectations.
>
> To support this, the client needs to remember the state sequence
> number associated with its layouts.  (By analogy, I think with
> the session sequence information you need to remember all of that.)
> This lets the clients ignore stale responses.
> I think this holds water even if layout ranges are subdivided.
>
>   
>>>> [email-masked] said:
>>>>         
>  > 
>  > Brent---
>  > This is a good analysis of the problem, but I'm not sure I
>  > agree that the "state sequence" view will completely address
>  > all state synchronization problems by itself.  Even with
>  > such a mechanism, I think you will still need to introduce
>  > some serialization at the server.  See my comments
>  > on your example below...  
>  > 
>  > -Steve
>  > 
>  > > From: Brent Welch [mailto:[email-masked]] 
>  > > Sent: Tuesday, June 27, 2006 6:03 PM
>  > > To: Benny Halevy
>  > > Cc: Talpey, Thomas; [email-masked]
>  > > Subject: Re: [nfsv4] pnfs: keeping the client and server 
>  > > layouts in sync 
>  > > 
>  > > This is a long, interesting thread.  Unfortunately for you
readers,
>  > > I seem to have generated a long addition to it.  The executive
> summary
>  > > is:
>  > >   I talk about ordering and deadlock.
>  > >   I talk about a state generation model to reason about the
> problem.
>  > >   I discuss if using session sequence information is a good way
>  > >     to represent the state generation model.
>  > >   I conclude that using session sequence information can be
> correct,
>  > >     but I would prefer a cleaner protocol that makes the state
>  > >     generation explicit.
>  > > 
>  > 
>  > <snip>
>  > 
>  > 
>  > > Suppose the server had a layout state generation number
>  > > (ok, some would call that a stateid) that incremented:
>  > > on each LAYOUTGET reply
>  > > on each LAYOUTRETURN reply
>  > > and was transmitted to the client on each CB_LAYOUTRECALL.
>  > > In this model the server owns that generation number, and
>  > > there is just one no matter how many clients there are.
>  > > The server does not need to keep a lock held on the file
>  > > while the RECALL is in-flight.  It only keeps a lock to
>  > > serialize updates to the state generation number and
>  > > generation of the replies or recall requests.
>  > > 
>  > > The function of the generation number is so the client can
>  > > order the layout operations.  Suppose it did a LAYOUTGET
>  > > that returned (0,1MB) at time 1.  Now suppose two things
>  > > happen concurrently, another LAYOUTGET for (1MB,2MB)
>  > > and a CB_LAYOUTRECALL for (0,1MB).  The server will order
>  > > those two, one way or the other, and transmit them to the client
>  > > that can put them back in order and process them.  Going
>  > > back to my contrived example, the server might decide
>  > > that the CB_LAYOUTRECALL for (0,1MB) occured at time 2,
>  > > and then it replied to the LAYOUTGET for (1MB,2MB) at
>  > > time 3, but passed a wider range for (0,2MB).  Even though
>  > > it appears goofy that the server gave the client a layout
>  > > for a range it just recalled, the sequence number makes
>  > > it crystal clear what the client should do.  The client owns
>  > > the layout from (0,2MB).
>  > > 
>  > > As an aside, the client can pass the server its last view
>  > > of the state sequence number, but the server doesn't depend
>  > > on that for correctness. 
>  > 
>  > We can conclude the server's intent by examining both replies
>  > and the generation numbers in the reply, but how does the client
>  > proceed in the absence of one of the replies?  For instance,
>  > suppose the client receives the CB_LAYOUTRECALL response
>  > with time "2".  Since time "1" was in the last reply the client
>  > got from the server, it can conclude that there are no intervening
>  > replies that have been mis-ordered, so it can start processing
>  > this recall.  Processing this recall means sending a LAYOUTRETURN
>  > for (0,1MB).  This conflicts with the reply to the LAYOUTGET that
>  > is in-flight, but which the client has not yet seen.  When the
>  > server receives the client's LAYOUTRETURN for (0,1MB) what should
>  > it do?  I can think of at least two options, neither of which
>  > seems satisfactory:
>  > 
>  > 1) If the client provides the latest gen number with the LAYOUT-
>  > RETURN ("2"), the server can recognize that this LAYOUTRETURN
>  > is not based on the latest server state.  Given this knowledge
>  > the server can reject the request with some sort of new error
>  > message, prompting the client to wait for a reply with the next
>  > gen number to get in sync with the server.  This approach has the
>  > disadvantage that it breaks our current defined semantics for
>  > the LAYOUTRETURN process, and relies on the client providing
>  > the latest gen number for correctness.
>  > 
>  > 2) It could process the layout return, and send a new reply with
>  > a new gen number "4".  Now if the client is required to process
>  > replies from the server is strict gen-number order, then the
>  > client can sync up its state with the server by processing the
>  > reply to the LAYOUTGET ("I've got 0,2MB") first, followed by the
>  > reply to the LAYOURETURN ("I don't have 0,1MB").  The disadvantage
>  > of this approach is that the client is left with (1,2MB) when
>  > the server was trying to give it (0,2MB).  Both client and
>  > server agree on the state---but its not the state that the
>  > server wanted!
>  > 
>  > Which leads me to the simplest solution:  after issuing the
>  > CB_LAYOUTRECALL, the server could reject LAYOUTGETS from the
>  > client until the recall processing is finished.  Then the server
>  > can grant the client (0,2MB) reliably, without the returns
>  > getting in the way.  Essentially, this is the "do one thing at
>  > a time" approach.
>  > 
>  > My point here is that while the "state sequence" view allows
>  > us to unambigously reason about the correct shared state view
>  > for any arbitrary interaction, the correct state may not be the
>  > state that we actually desire to achieve.  I think we will always
>  > need a combination of serialization to reduce the number of
possible
>  > interactions that we have to consider, and "state sequencing"
>  > to resolve the inevitable races that occur in the interactions
>  > we *do* want to consider.
>  > 
>  > 
>  > <remaining thread snipped>
>  > 
>  > 
>
> --
> Brent Welch
> Software Architect, Panasas Inc
> Accelerating Time to Results(tm) with Clustered Storage
>
> www.panasas.com
> [email-masked]
>
>
>