Subject: RE: [nfsv4] pnfs: keeping the client and server layouts in sync

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 29 Jun 2006 06:27:39 -0400

> I'm sorry to cloud the issue with another mechanism.  My overall 
> point is that "welch's state generation" and "NFSv4 stateids" and 
> "CB_SEQUENCE information" should be isomorphic.  

If they were isomorphic then their characteristics would be pretty
much identical.  

> They are all very similar mechanisms proposed to solve the same 
> problem. 

They certainly have similarities but they also have differences
which is why this thread lives on. 

> Personally, I can reason about a server-maintained state generation 
> number. 

Right but "I can reason about X" is not very helpful in reaching a
consensus if somebody else says "I can reason about Y".  How do we
then choose between X and Y?

> If you say that stateids had problems with recovery, then we 
> should look for similar problems in the use of my state generation 
> or the use of session sequence information.  

I was referring not to stateids but to sequence id's associated with
lockowners, which was the big mistake in v4.0 that we are only fixing
in v4.1 (my personal opinion).  Sessions doesn't have those problems
(lifetime of owners) because you create a session, every sequence for
each slotid starts at zero and we separate the quasi-transport aspect
of session from anything application-specific whose lifetime requirements
might be different.

> I believe we handle it by revoking all the layout information.  I 
> would agree that any server-crash recovery requires a message exchange 
> so the client can resync.

Perhaps so, but until you make a proposal including stuff to address
these issues, nobody really knows for sure.

This gets to what is bothering me about this.  For several months, we have
been assuming that sessions plus some sort of CB_SEQUENCE addition
was the preferred way to do this.  Getting to proposed text took a lot
more time than we expected because a number of people have demanding
jobs, and their time was not available.  But there was ample opportunity
to pursue alternate mechanisms during that time.  We didn't hear anything
about them.  Now, I think a new mechanism is going to lengthen the v4.1
schedule since the details really haven't been worked out.  The question
is why should we change at this point and lengthen the schedule.  There
has to be something concrete to justify the change and I'm not hearing 
it.

> In contrast, what happens when a network partition eliminates the 
> sessions, but the server and client remain active?  

A network partition does not eliminate a session.  If it did, a lot 
of the value of sessions would be vitiated.  The idea is that when 
you get to reconnect the sequence values are as they were so that
you have exactly once semantics.

A network partition will result fairly soon in loss of a lease and
then possibly loss of lock state and revocation of layouts.  

> We need something in the spec to explain how that works. 

I assume that the spec makes this clear.  I've been in so many 
discussions about this that I can't read the spec with appropriately
naïve eyes to see if it really explains things.  If it doesn't,
we need to fix it.

> In a past life the state generation number was an attribute of the 
> file, so it was persistent.  Both/either the client or server could 
> crash and restart, and they could validate any cached information 
> quite simply.  NFSv4 stateids are per-client/per-file (at least!) 
> so there are a lot more of them and it is more expensive to make 
> them persistent.

A per-file persistent quantity would work.  Is that what you are
proposing?  I have no problem implementing that, but in the past,
requirements for additional on-disk state to implement a protocol
feature have trouble getting consensus.  That's the thing about
proposing an entirely new mechanism, even though in some sense
similar.  You have to get consensus all over again.  Also, I have
heard on this thread that some people want a compact (and thus
per-client) sequence, which has yet a different set of issues.
Again, the sequence approoach is not yet a concrete proposal and
it will take considerable time to get to the stage of concrete
proposal.

> > What specifically is broken with the following:
> > 
> >      Use a do-one-thing-at-a-time model for voluntary layout
> >      returns, requiring LAYOUTGET's for the same file to be
> >      resolved before proceeding to do the voluntary LAYOUTRETURN.

> I think everyone agrees on the above rule.

And therefore you agree with rule.  Right?

> >      For recall-prompted LAYOUTRETURN allow you to synchronize only
> >      with LAYOUTGETs listed in the CB_SEQUENCE, with the understanding
> >      that [the server] will not grant any subsequent LAYOUTGETs (those not
> >      in the CB_SEQUENCE) any region that overlaps that recalled.

> Uh, for how long?

Until the recall is complete, i.e. the client returns the entire range 
being recalled.  

> We need more precise language for this rule.  

I think David Black has proposed sufficiently precise language.  If not,
we should fix it.

> Benny's original thread was about this "meta operation" that spanned 
> the time of the server's recall to the final RETURN done by the client.

> We need to define what you mean by "will not grant a LAYOUTGET"
> (NFS4_RETRY ? NFS4_LAYOUTDENIED?)

I should have said:  "Will not grant a layout in response to a LAYOUTGET
operation."

> and we need to bound the time during which the server doesn't grant a 
> layout that it has just recalled.

As I said, until the recall is complete.

> I will refer again to my example where the client does concurrent 
> RETURN and GET for the same range.  

I don't understand.  Several paragraphs I said that part of the solution
is to impose a restriction that you don't do RETURN and GET on the same
file at the same time.  I thought you agreed to that.  Maybe I wasn't
clear.  I mentioned the case where a GET was done first.  I think the
same has to apply when the RETURN is first.  A one-thing-at-a-time
restriction per-file is what I'm proposing here, except when recalls
are involved, so I'm ruling this case out of bounds.  You don't have 
to agree with that but if you don't, I think you should explain why 
that is not a tolerable restriction.

> I don't see how the session sequence numbers solve that problem.  

Clearly it doesn't.  That's why the synchronization constraint cited
above has to be added to it.

> The state generation number doesn't make any rules about what the 
> server does, but it makes the server's serialization explicit so 
> the client knows how to respond.

I understand that.  That is why these are not isomorphic even though
somewhat similar.  The question is why is your approach better and
enough better to justify switching at this point?

> If you are going to impose a rule that the client has to issue 
> all GET and RETURN operations over the same session slot, then 
> you'll have to open up an wider interface to the RPC transport 
> layer.

Not proposing any such thing.  

> Again, in a past life I implemented RPC sessions and they were 
> hidden below the RPC transport interface.  I didn't have any 
> higher level protocols that knew about sessions.

So we have a situation were delegation recall does know about 
sessions and so far nobody really has had an issue with that and
up until now we had been assuming the same about layout recall.  
I still don't see what the problem is.

> heh - I view a single state generation number as simpler than 
> a concatenation of session sequence numbers.  

As a single object in isolation sure.  But in terms of the 
protocol as a whole, it is more complicated because you are 
adding another piece of sequencing state (application specific)
to the basic sessions sequence state.

> I think you view it as more complicated because there is an 
> implication of persistence during recovery.

I don't think that adds protocol complexity.  I think it clearly
does add complexity (and time) to the process of closing on the 
structure of the protocol.  That kind of delay can of course be 
justified, if it results in a better protocol, but I haven't 
heard arguments that it will result in a better protocol, only 
one that is different.

> Again, I'm not hung up on having a state generation number as 
> the implementation, but I feel pretty strongly that it is a good 
> model for what happens in the distributed system.  If you can 
> map the CB_SEQUENCE implementation to an abstract model of a 
> state generation number, then I'd be confident it was correct 
> and complete.

In some sense, the CB_SEQUENCE approach is not complete.  It 
limits the need for sequencing mechanisms by restricting the 
set of possible racing messages to a minimum.  I don't think
that is problematic and what I would be looking for to justify
a change is some reason to believe that that either will
not work is overly restrictive and so far nobody has provided
that.  What I'm hearing is basically personal preference and
if everybody insisted on their own personal preferences in 
these matters, the working group would not make much progress.

> No matter what you do, we need a concrete rule for what happens 
> during the interval between the RECALL and the subsequence RETURNS 
> and GETs done by clients.

Until the recall is complete, the server does not do grants to the
client that overlap the range recalled. 

Clients never have GETs and RETURNs outstanding simultaneously for
the same file except in the case of RETURNS prompted by recalls in
which case the CB_SEQUENCE identifies the GETs that have to complete
before the RETURN and others are OK.

So let's look at the mapping between that and the sequence approach.
First note that if I only have one operation outstanding for a given
file, then I would always see an ascending sequence number but there 
is no need for a sequence number to be sent.  So we need to consider
all the cases in which, we are more liberal than a pure one-thing-
at-a-time approach to see whether people can get out of sync.

First observe that the total ordering of a sequence number is in some
sense an approximation to the real data, which is a partial order.
If I send two non-overlapping operations, the server may choose to
process them in parallel with there no being reason to define either
one as happening before the other, until you get to a point where
you have to send sequence values (if you do) and you arbitrarily
choose one as happening before the other.  The point here is that
the client needs a partial ordering  of overlapping operations to
avoid getting out of sync and the server's execution follows that
partial ordering, but that sequence number is a simple device to
transmit the partial ordering.  It overspecifies the reality but
that overspecification is harmless, as long as the total order
of the sequence values is consistent with the partial order.

So, if we could be assured that no overlapping GETs and RETURNs
were outstanding then the partial order would be clear, since it
would be the client's execution order and again there would be
no need for a sequence number.  One of the issues we have to 
deal with is that generally the client can't determine what 
may overlap and the first step of dealing with that is, for
most case not to allow GETs and RETURNs to be outstanding at
the same time.

Note that that does still allow multiple GETs or multiple
RETURNs to be outstanding at the same, but I assert that that
also is harmless.  If the server processes them in one order
and I process them in a different order, then we still arrive
at the same result.  

So now proceed to the exception for RETURNs prompted by recalls
and how that maps to a sequence.  So suppose that the server
included in the CB_SEQUENCE all of the GETs from the client 
that he had in fact executed at the time of the recall.  Then
that set of GETs would identify his state sequence with respect 
to that client and the client by issuing recalls at that point 
would be consistent with the server's view of things and so 
would not get out of sync.  Logically the client could consider
any GETs not mentioned as having sequence values greater than
the RETURNs.  So I think that the client and server are prevented
from getting out of sync.  Now consider an optimization of that
in which the server excludes GETs that may have been executed
but where the layout granted does not overlap the recall.  That
violates the state sequence model but in harmless way since
the ranges overlapping,  The client's order and the server's
may different but both are compatible with the partial order that
shows execution ordering on overlapping ranges.

-----Original Message-----
From: Brent Welch [mailto:[email-masked]] 
Sent: Wednesday, June 28, 2006 6:35 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]; Talpey, Thomas; [email-masked]
Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in sync 

>>>"Noveck, Dave" said:
 >
 > As someone who went through the experience of v4.0 and has  > regretted the decision, which seemed very natural at the time,  > to add application-specific sequencing, let me insert a bit of  > a cautionary note here.  When you propose an application-specific  > sequencing mechanism, it is very easy to put together simple  > examples that show how such a thing would work, and effectively  > ignore the problem that often makes such an approach problematic.

I'm sorry to cloud the issue with another mechanism.  My overall point is that "welch's state generation" and "NFSv4 stateids" and "CB_SEQUENCE information" should be isomorphic.  They are all very similar mechanisms proposed to solve the same problem.  Personally, I can reason about a server-maintained state generation number.

If you say that stateids had problems with recovery, then we should look for similar problems in the use of my state generation or the use of session sequence information.  I believe we handle it by revoking all the layout information.  I would agree that any server-crash recovery requires a message exchange so the client can resync.

In contrast, what happens when a network partition eliminates the sessions, but the server and client remain active?  We need something in the spec to explain how that works.

 > In the steady state, there is an object for which the client and  > the server each have a sequence value and that sequence value  > provides a very helpful ordering.  Tastes great and less filling.
 > The problems that are difficult turn out to be those outside that  > steady state.  What happen if an object is new to the server but  > not to the client, or vice versa.  How do you avoid having to  > keep such objects around precisely because client and server have  > a sequence value that they share and it might troublesome if one  > of the two were to drop his.  That's how we got OPEN_CONFIRM and  > RELEASE_LOCKOWNER.  These are ops that we were forced to create  > to deal consequences of a simple per-owner sequence number.

In a past life the state generation number was an attribute of the file, so it was persistent.  Both/either the client or server could crash and restart, and they could validate any cached information quite simply.  NFSv4 stateids are per-client/per-file (at least!) so there are a lot more of them and it is more expensive to make them persistent.

 > I don't know for sure that analogous problems would exist for  > this sort of sequencing but I don't think one should assume that  > they don't exist either.  Before we go down this routine, I'd like  > to understand the motivation.
 >
 > What specifically is broken with the following:
 > 
 >      Use a do-one-thing-at-a-time model for voluntary layout
 >      returns, requiring LAYOUTGET's for the same file to be
 >      resolved before proceeding to do the voluntary LAYOUTRETURN.

I think everyone agrees on the above rule.

 >      For recall-prompted LAYOUTRETURN allow you to synchronize only
 >      with LAYOUTGETs listed in the CB_SEQUENCE, with the understanding
 >      that [the server] will not grant any subsequent LAYOUTGETs (those not
 >      in the CB_SEQUENCE) any region that overlaps that recalled.

Uh, for how long?  We need more precise language for this rule.  Benny's original thread was about this "meta operation" that spanned the time of the server's recall to the final RETURN done by the client.
We need to define what you mean by "will not grant a LAYOUTGET"
(NFS4_RETRY ? NFS4_LAYOUTDENIED?), and we need to bound the time during which the server doesn't grant a layout that it has just recalled.

I will refer again to my example where the client does concurrent RETURN and GET for the same range.  I don't see how the session sequence numbers solve that problem.  The state generation number doesn't make any rules about what the server does, but it makes the server's serialization explicit so the client knows how to respond.

If you are going to impose a rule that the client has to issue all GET and RETURN operations over the same session slot, then you'll have to open up an wider interface to the RPC transport layer.
Again, in a past life I implemented RPC sessions and they were hidden below the RPC transport interface.  I didn't have any higher level protocols that knew about sessions.

 > I understand that Benny has some uneasiness about the correctness  > of synchronization with this approach, but I haven't seen an example  > presented where this approach has the client and server get out of  > sync or where there is another significant negative consequence.
 >
 > Before we create another sequencing mechanism (which may wind being  > more complicated than it first appears), I'd like to be clear on the  > motivation.  Why is it that a simpler approach won't work or is  > otherwise undesirable?

heh - I view a single state generation number as simpler than a concatenation of session sequence numbers.  I think you view it as more complicated because there is an implication of persistence during recovery.

Again, I'm not hung up on having a state generation number as the implementation, but I feel pretty strongly that it is a good model for what happens in the distributed system.  If you can map the CB_SEQUENCE implementation to an abstract model of a state generation number, then I'd be confident it was correct and complete.

No matter what you do, we need a concrete rule for what happens during the interval between the RECALL and the subsequence RETURNS and GETs done by clients.

 > -----Original Message-----
 > From: Brent Welch [mailto:[email-masked]]  > Sent: Wednesday, June 28, 2006 12:21 PM  > To: [email-masked]  > Cc: [email-masked]; Talpey, Thomas; [email-masked]  > Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in =  > sync  >  >  >  > [email-masked] said:
 > > I think you will still need to introduce some serialization at the  > server.
 >
 > The sequence number represents the serialization that has occurred.
 >
 > Benny points out that it is the ordering between LAYOUTGET and  > LAYOUTRETURN that is most important. The RECALL is a kind of  > hint to the client that it is supposed to do something.  (Of  > course, if you ignore the hints, the server will fence you.)  > So, let's add LAYOUTRETURN to these examples:
 >
 > LAYOUTGET(0,1MB) at time 1
 > CB_LAYOUTRECALL(0,1MB) at time 2
 > LAYOUTRETURN(0,1MB) at time X
 > LAYOUTGET(1MB,2MB) at time Y, but returning (0,2MB)  >  > if (X < Y), then the server processed the return before the get,  > and the client owns (0,2MB).  This scenario is that the server  > wanted to do a recall, got the return, but then "changed its mind"
 > and decided to give the layout back to the client.
 >
 > If (Y < X), then the server processed the get first and then  > the return, so the client owns (1MB,2MB).  This matches our  > expectations.
 >
 > To support this, the client needs to remember the state sequence  > number associated with its layouts.  (By analogy, I think with  > the session sequence information you need to remember all of that.)  > This lets the clients ignore stale responses.
 > I think this holds water even if layout ranges are subdivided.
 >
 > >>>[email-masked] said:
 >  >
 >  > Brent---
 >  > This is a good analysis of the problem, but I'm not sure I  >  > agree that the "state sequence" view will completely address  >  > all state synchronization problems by itself.  Even with  >  > such a mechanism, I think you will still need to introduce  >  > some serialization at the server.  See my comments  >  > on your example below... 
 >  >
 >  > -Steve
 >  >
 >  > > -----Original Message-----
 >  > > From: Brent Welch [mailto:[email-masked]]  >  > > Sent: Tuesday, June 27, 2006 6:03 PM  >  > > To: Benny Halevy  >  > > Cc: Talpey, Thomas; [email-masked]  >  > > Subject: Re: [nfsv4] pnfs: keeping the client and server  >  > > layouts in sync  >  > >  >  > > This is a long, interesting thread.  Unfortunately for you =  > readers,  >  > > I seem to have generated a long addition to it.  The executive  > summary  >  > > is:
 >  > >   I talk about ordering and deadlock.
 >  > >   I talk about a state generation model to reason about the
 > problem.
 >  > >   I discuss if using session sequence information is a good way
 >  > >     to represent the state generation model.
 >  > >   I conclude that using session sequence information can be
 > correct,
 >  > >     but I would prefer a cleaner protocol that makes the state
 >  > >     generation explicit.
 >  > >
 >  >
 >  > <snip>
 >  >
 >  >
 >  > > Suppose the server had a layout state generation number  >  > > (ok, some would call that a stateid) that incremented:
 >  > > on each LAYOUTGET reply
 >  > > on each LAYOUTRETURN reply
 >  > > and was transmitted to the client on each CB_LAYOUTRECALL.
 >  > > In this model the server owns that generation number, and  >  > > there is just one no matter how many clients there are.
 >  > > The server does not need to keep a lock held on the file  >  > > while the RECALL is in-flight.  It only keeps a lock to  >  > > serialize updates to the state generation number and  >  > > generation of the replies or recall requests.
 >  > >
 >  > > The function of the generation number is so the client can  >  > > order the layout operations.  Suppose it did a LAYOUTGET  >  > > that returned (0,1MB) at time 1.  Now suppose two things  >  > > happen concurrently, another LAYOUTGET for (1MB,2MB)  >  > > and a CB_LAYOUTRECALL for (0,1MB).  The server will order  >  > > those two, one way or the other, and transmit them to the client  >  > > that can put them back in order and process them.  Going  >  > > back to my contrived example, the server might decide  >  > > that the CB_LAYOUTRECALL for (0,1MB) occured at time 2,  >  > > and then it replied to the LAYOUTGET for (1MB,2MB) at  >  > > time 3, but passed a wider range for (0,2MB).  Even though  >  > > it appears goofy that the server gave the client a layout  >  > > for a range it just recalled, the sequence number makes  >  > > it crystal clear what the client should do.  The client owns  >  > > the layout from (0,2MB).
 >  > >
 >  > > As an aside, the client can pass the server its last view  >  > > of the state sequence number, but the server doesn't depend  >  > > on that for correctness.
 >  >
 >  > We can conclude the server's intent by examining both replies  >  > and the generation numbers in the reply, but how does the client  >  > proceed in the absence of one of the replies?  For instance,  >  > suppose the client receives the CB_LAYOUTRECALL response  >  > with time "2".  Since time "1" was in the last reply the client  >  > got from the server, it can conclude that there are no intervening  >  > replies that have been mis-ordered, so it can start processing  >  > this recall.  Processing this recall means sending a LAYOUTRETURN  >  > for (0,1MB).  This conflicts with the reply to the LAYOUTGET that  >  > is in-flight, but which the client has not yet seen.  When the  >  > server receives the client's LAYOUTRETURN for (0,1MB) what should  >  > it do?  I can think of at least two options, neither of which  >  > seems satisfactory:
 >  >
 >  > 1) If the client provides the latest gen number with the LAYOUT-  >  > RETURN ("2"), the server can recognize that this LAYOUTRETURN  >  > is not based on the latest server state.  Given this knowledge  >  > the server can reject the request with some sort of new error  >  > message, prompting the client to wait for a reply with the next  >  > gen number to get in sync with the server.  This approach has the  >  > disadvantage that it breaks our current defined semantics for  >  > the LAYOUTRETURN process, and relies on the client providing  >  > the latest gen number for correctness.
 >  >
 >  > 2) It could process the layout return, and send a new reply with  >  > a new gen number "4".  Now if the client is required to process  >  > replies from the server is strict gen-number order, then the  >  > client can sync up its state with the server by processing the  >  > reply to the LAYOUTGET ("I've got 0,2MB") first, followed by the  >  > reply to the LAYOURETURN ("I don't have 0,1MB").  The disadvantage  >  > of this approach is that the client is left with (1,2MB) when  >  > the server was trying to give it (0,2MB).  Both client and  >  > server agree on the state---but its not the state that the  >  > server wanted!
 >  >
 >  > Which leads me to the simplest solution:  after issuing the  >  > CB_LAYOUTRECALL, the server could reject LAYOUTGETS from the  >  > client until the recall processing is finished.  Then the server  >  > can grant the client (0,2MB) reliably, without the returns  >  > getting in the way.  Essentially, this is the "do one thing at  >  > a time" approach.
 >  >
 >  > My point here is that while the "state sequence" view allows  >  > us to unambigously reason about the correct shared state view  >  > for any arbitrary interaction, the correct state may not be the  >  > state that we actually desire to achieve.  I think we will always  >  > need a combination of serialization to reduce the number of possible  >  > interactions that we have to consider, and "state sequencing"
 >  > to resolve the inevitable races that occur in the interactions  >  > we *do* want to consider.
 >  >
 >  >

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brent Welch <[email-masked]>
Date: Fri, 30 Jun 2006 09:31:33 -0700

"OK" - thanks for the detailed reply.

I agree that CB_SEQUENCE, coupled with the rule that the
client does not issue more GETS until it has done all
the RETURNS implied by a RECALL, is necessary and sufficient
to keep the client and server in sync.  And, by "RETURNS are done"
I mean that the server's reply to those operations have been
received by the client.  Also, and I believe this is in your
rules, if a client is doing voluntary RETURNS, those should
complete before it issues more GETS.  Concurrent GETS are OK.
Concurrent RETURNS are OK.

Again, I floated the state sequence model as a way to think
about the problem, and I am perfectly agreable to keeping
the existing CB_SEQUENCE mechanism.  This is an important
issue within the protocol and I really wanted to make sure
we teased out all the fine points and clarified them.

 > na=EFve eyes to see if it really explains things.  If it doesn't,
 > we need to fix it.
 > 
 >  > As someone who went through the experience of v4.0 and has  > 
 > regretted the decision, which seemed very natural at the time,  > to 
 > add application-specific sequencing, let me insert a bit of  > a 
 > cautionary note here.  When you propose an application-specific  > 
 > sequencing mechanism, it is very easy to put together simple  > 
 > examples that show how such a thing would work, and effectively  > 
 > ignore the problem that often makes such an approach problematic.
 > 
 > I'm sorry to cloud the issue with another mechanism.  My overall point 
 > is that "welch's state generation" and "NFSv4 stateids" and 
 > "CB_SEQUENCE information" should be isomorphic.  They are all very 
 > similar mechanisms proposed to solve the same problem.  Personally, I 
 > can reason about a server-maintained state generation number.
 > 
 > If you say that stateids had problems with recovery, then we should 
 > look for similar problems in the use of my state generation or the use 
 > of session sequence information.  I believe we handle it by revoking 
 > all the layout information.  I would agree that any server-crash 
 > recovery requires a message exchange so the client can resync.
 > 
 > sessions, but the server and client remain active?  We need something 
 > in the spec to explain how that works.
 > 
 >  > In the steady state, there is an object for which the client and  > 
 > the server each have a sequence value and that sequence value  > 
 > provides a very helpful ordering.  Tastes great and less filling.
 >  > The problems that are difficult turn out to be those outside that  > 
 > steady state.  What happen if an object is new to the server but  > not 
 > to the client, or vice versa.  How do you avoid having to  > keep such 
 > objects around precisely because client and server have  > a sequence 
 > value that they share and it might troublesome if one  > of the two 
 > were to drop his.  That's how we got OPEN_CONFIRM and  > 
 > RELEASE_LOCKOWNER.  These are ops that we were forced to create  > to 
 > deal consequences of a simple per-owner sequence number.
 > 
 > crash and restart, and they could validate any cached information quite 
 > simply.  NFSv4 stateids are per-client/per-file (at least!) so there 
 > are a lot more of them and it is more expensive to make them 
 > persistent.
 > 
 >  > I don't know for sure that analogous problems would exist for  > 
 > this sort of sequencing but I don't think one should assume that  > 
 > they don't exist either.  Before we go down this routine, I'd like  > 
 > to understand the motivation.
 >  >
 > Uh, for how long?  We need more precise language for this rule.  
 > (NFS4_RETRY ? NFS4_LAYOUTDENIED?), and we need to bound the time during 
 > which the server doesn't grant a layout that it has just recalled.
 > 
 > RETURN and GET for the same range.  I don't see how the session 
 > sequence numbers solve that problem.  The state generation number 
 > doesn't make any rules about what the server does, but it makes the 
 > server's serialization explicit so the client knows how to respond.
 > 
 > If you are going to impose a rule that the client has to issue all GET 
 > and RETURN operations over the same session slot, then you'll have to 
 > open up an wider interface to the RPC transport layer.
 > Again, in a past life I implemented RPC sessions and they were hidden 
 > below the RPC transport interface.  I didn't have any higher level 
 > protocols that knew about sessions.
 > 
 >  > I understand that Benny has some uneasiness about the correctness  > 
 > of synchronization with this approach, but I haven't seen an example  > 
 > presented where this approach has the client and server get out of  > 
 > sync or where there is another significant negative consequence.
 >  >
 >  > Before we create another sequencing mechanism (which may wind being  
 > > more complicated than it first appears), I'd like to be clear on the  
 > > motivation.  Why is it that a simpler approach won't work or is  > 
 > otherwise undesirable?
 > 
 > heh - I view a single state generation number as simpler than a 
 > concatenation of session sequence numbers.  I think you view it as more 
 > complicated because there is an implication of persistence during 
 > recovery.
 > 
 > Again, I'm not hung up on having a state generation number as the 
 > implementation, but I feel pretty strongly that it is a good model for 
 > what happens in the distributed system.  If you can map the CB_SEQUENCE 
 > implementation to an abstract model of a state generation number, then 
 > I'd be confident it was correct and complete.
 > 
 > No matter what you do, we need a concrete rule for what happens during 
 > the interval between the RECALL and the subsequence RETURNS and GETs 
 > done by clients.
 > 
 >  > From: Brent Welch [mailto:[email-masked]]  > Sent: Wednesday, 
 > June 28, 2006 12:21 PM  > To: [email-masked]  > Cc: 
 > [email-masked]; Talpey, Thomas; [email-masked]  > Subject: Re: 
 > [nfsv4] pnfs: keeping the client and server layouts in =3D  > sync  >  
 >  > > I think you will still need to introduce some serialization at the 
 >  > server.
 >  >
 >  > Benny points out that it is the ordering between LAYOUTGET and  > 
 > LAYOUTRETURN that is most important. The RECALL is a kind of  > hint to 
 > the client that it is supposed to do something.  (Of  > course, if you 
 > ignore the hints, the server will fence you.)  > So, let's add 
 > LAYOUTRETURN to these examples:
 >  >
 >  > LAYOUTGET(1MB,2MB) at time Y, but returning (0,2MB)  >  > if (X < 
 > Y), then the server processed the return before the get,  > and the 
 > client owns (0,2MB).  This scenario is that the server  > wanted to do 
 > a recall, got the return, but then "changed its mind"
 >  > If (Y < X), then the server processed the get first and then  > the 
 > return, so the client owns (1MB,2MB).  This matches our  > 
 > expectations.
 >  >
 >  > To support this, the client needs to remember the state sequence  > 
 > number associated with its layouts.  (By analogy, I think with  > the 
 > session sequence information you need to remember all of that.)  > This 
 > lets the clients ignore stale responses.
 >  >  > This is a good analysis of the problem, but I'm not sure I  >  > 
 > agree that the "state sequence" view will completely address  >  > all 
 > state synchronization problems by itself.  Even with  >  > such a 
 > mechanism, I think you will still need to introduce  >  > some 
 > serialization at the server.  See my comments  >  > on your example 
 > below...
 >  >  >
 >  >  > -Steve
 >  >  >
 >  >  > > From: Brent Welch [mailto:[email-masked]]  >  > > Sent: 
 > Tuesday, June 27, 2006 6:03 PM  >  > > To: Benny Halevy  >  > > Cc: 
 > Talpey, Thomas; [email-masked]  >  > > Subject: Re: [nfsv4] pnfs: 
 > keeping the client and server  >  > > layouts in sync  >  > >  >  > > 
 > This is a long, interesting thread.  Unfortunately for you =3D  > 
 > readers,  >  > > I seem to have generated a long addition to it.  The 
 > executive  > summary  >  > > is:
 >  >  > > Suppose the server had a layout state generation number  >  > > 
 > (ok, some would call that a stateid) that incremented:
 >  >  > > on each LAYOUTGET reply
 >  >  > > In this model the server owns that generation number, and  >  > 
 > > there is just one no matter how many clients there are.
 >  >  > > The server does not need to keep a lock held on the file  >  > 
 > > while the RECALL is in-flight.  It only keeps a lock to  >  > > 
 > serialize updates to the state generation number and  >  > > generation 
 > of the replies or recall requests.
 >  >  > >
 >  >  > > The function of the generation number is so the client can  >  
 > > > order the layout operations.  Suppose it did a LAYOUTGET  >  > > 
 > that returned (0,1MB) at time 1.  Now suppose two things  >  > > happen 
 > concurrently, another LAYOUTGET for (1MB,2MB)  >  > > and a 
 > CB_LAYOUTRECALL for (0,1MB).  The server will order  >  > > those two, 
 > one way or the other, and transmit them to the client  >  > > that can 
 > put them back in order and process them.  Going  >  > > back to my 
 > contrived example, the server might decide  >  > > that the 
 > CB_LAYOUTRECALL for (0,1MB) occured at time 2,  >  > > and then it 
 > replied to the LAYOUTGET for (1MB,2MB) at  >  > > time 3, but passed a 
 > wider range for (0,2MB).  Even though  >  > > it appears goofy that the 
 > server gave the client a layout  >  > > for a range it just recalled, 
 > the sequence number makes  >  > > it crystal clear what the client 
 > should do.  The client owns  >  > > the layout from (0,2MB).
 >  >  > >
 >  >  > > As an aside, the client can pass the server its last view  >  > 
 > > of the state sequence number, but the server doesn't depend  >  > > 
 > on that for correctness.
 >  >  >
 >  >  > We can conclude the server's intent by examining both replies  >  
 > > and the generation numbers in the reply, but how does the client  >  
 > > proceed in the absence of one of the replies?  For instance,  >  > 
 > suppose the client receives the CB_LAYOUTRECALL response  >  > with 
 > time "2".  Since time "1" was in the last reply the client  >  > got 
 > from the server, it can conclude that there are no intervening  >  > 
 > replies that have been mis-ordered, so it can start processing  >  > 
 > this recall.  Processing this recall means sending a LAYOUTRETURN  >  > 
 > for (0,1MB).  This conflicts with the reply to the LAYOUTGET that  >  > 
 > is in-flight, but which the client has not yet seen.  When the  >  > 
 > server receives the client's LAYOUTRETURN for (0,1MB) what should  >  > 
 > it do?  I can think of at least two options, neither of which  >  > 
 > seems satisfactory:
 >  >  >
 >  >  > 1) If the client provides the latest gen number with the LAYOUT-  
 > >  > RETURN ("2"), the server can recognize that this LAYOUTRETURN  >  
 > > is not based on the latest server state.  Given this knowledge  >  > 
 > the server can reject the request with some sort of new error  >  > 
 > message, prompting the client to wait for a reply with the next  >  > 
 > gen number to get in sync with the server.  This approach has the  >  > 
 > disadvantage that it breaks our current defined semantics for  >  > the 
 > LAYOUTRETURN process, and relies on the client providing  >  > the 
 > latest gen number for correctness.
 >  >  >
 >  >  > 2) It could process the layout return, and send a new reply with  
 > >  > a new gen number "4".  Now if the client is required to process  > 
 >  > replies from the server is strict gen-number order, then the  >  > 
 > client can sync up its state with the server by processing the  >  > 
 > reply to the LAYOUTGET ("I've got 0,2MB") first, followed by the  >  > 
 > reply to the LAYOURETURN ("I don't have 0,1MB").  The disadvantage  >  
 > > of this approach is that the client is left with (1,2MB) when  >  > 
 > the server was trying to give it (0,2MB).  Both client and  >  > server 
 > agree on the state---but its not the state that the  >  > server 
 > wanted!
 >  >  >
 >  >  > Which leads me to the simplest solution:  after issuing the  >  > 
 > CB_LAYOUTRECALL, the server could reject LAYOUTGETS from the  >  > 
 > client until the recall processing is finished.  Then the server  >  > 
 > can grant the client (0,2MB) reliably, without the returns  >  > 
 > getting in the way.  Essentially, this is the "do one thing at  >  > a 
 > time" approach.
 >  >  >
 >  >  > My point here is that while the "state sequence" view allows  >  
 > > us to unambigously reason about the correct shared state view  >  > 
 > for any arbitrary interaction, the correct state may not be the  >  > 
 > state that we actually desire to achieve.  I think we will always  >  > 
 > need a combination of serialization to reduce the number of possible  > 
 >  > interactions that we have to consider, and "state sequencing"
 >  >  > to resolve the inevitable races that occur in the interactions  > 
 >  > we *do* want to consider.
 >  >  >
 >  >  >
 > 
 > --
 > Brent Welch

www.panasas.com
[email-masked]