Subject: RE: [nfsv4] pnfs: keeping the client and server layouts in sync

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 28 Jun 2006 13:44:54 -0400

As someone who went through the experience of v4.0 and has
regretted the decision, which seemed very natural at the time,
to add application-specific sequencing, let me insert a bit of
a cautionary note here.  When you propose an application-specific
sequencing mechanism, it is very easy to put together simple 
examples that show how such a thing would work, and effectively
ignore the problem that often makes such an approach problematic.

In the steady state, there is an object for which the client and
the server each have a sequence value and that sequence value
provides a very helpful ordering.  Tastes great and less filling.
The problems that are difficult turn out to be those outside that
steady state.  What happen if an object is new to the server but
not to the client, or vice versa.  How do you avoid having to
keep such objects around precisely because client and server have
a sequence value that they share and it might troublesome if one
of the two were to drop his.  That's how we got OPEN_CONFIRM and
RELEASE_LOCKOWNER.  These are ops that we were forced to create 
to deal consequences of a simple per-owner sequence number.

I don't know for sure that analogous problems would exist for
this sort of sequencing but I don't think one should assume that 
they don't exist either.  Before we go down this routine, I'd like
to understand the motivation.

What specifically is broken with the following:

     Use a do-one-thing-at-a-time model for voluntary layout
     returns, requiring LAYOUTGET's for the same file to be
     resolved before proceeding to do the voluntary LAYOUTRETURN.

     For recall-prompted LAYOUTRETURN allow you to synchronize only
     with LAYOUTGETs listed in the CB_SEQUENCE, with the understanding
     that client will not grant any subsequent LAYOUTGETs (those not
     in the CB_SEQUENCE) any region that overlaps that recalled.

I understand that Benny has some uneasiness about the correctness
of synchronization with this approach, but I haven't seen an example 
presented where this approach has the client and server get out of 
sync or where there is another significant negative consequence.

Before we create another sequencing mechanism (which may wind being
more complicated than it first appears), I'd like to be clear on the
motivation.  Why is it that a simpler approach won't work or is 
otherwise undesirable? 

-----Original Message-----
From: Brent Welch [mailto:[email-masked]] 
Sent: Wednesday, June 28, 2006 12:21 PM
To: [email-masked]
Cc: [email-masked]; Talpey, Thomas; [email-masked]
Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in sync

[email-masked] said:
> I think you will still need to introduce some serialization at the
server.

The sequence number represents the serialization that has occurred.

Benny points out that it is the ordering between LAYOUTGET and
LAYOUTRETURN that is most important. The RECALL is a kind of
hint to the client that it is supposed to do something.  (Of
course, if you ignore the hints, the server will fence you.)
So, let's add LAYOUTRETURN to these examples:

LAYOUTGET(0,1MB) at time 1
CB_LAYOUTRECALL(0,1MB) at time 2
LAYOUTRETURN(0,1MB) at time X
LAYOUTGET(1MB,2MB) at time Y, but returning (0,2MB)

if (X < Y), then the server processed the return before the get,
and the client owns (0,2MB).  This scenario is that the server
wanted to do a recall, got the return, but then "changed its mind"
and decided to give the layout back to the client.

If (Y < X), then the server processed the get first and then
the return, so the client owns (1MB,2MB).  This matches our
expectations.

To support this, the client needs to remember the state sequence
number associated with its layouts.  (By analogy, I think with
the session sequence information you need to remember all of that.)
This lets the clients ignore stale responses.
I think this holds water even if layout ranges are subdivided.

>>>[email-masked] said:
 > 
 > Brent---
 > This is a good analysis of the problem, but I'm not sure I
 > agree that the "state sequence" view will completely address
 > all state synchronization problems by itself.  Even with
 > such a mechanism, I think you will still need to introduce
 > some serialization at the server.  See my comments
 > on your example below...  
 > 
 > -Steve
 > 
 > > -----Original Message-----
 > > From: Brent Welch [mailto:[email-masked]] 
 > > Sent: Tuesday, June 27, 2006 6:03 PM
 > > To: Benny Halevy
 > > Cc: Talpey, Thomas; [email-masked]
 > > Subject: Re: [nfsv4] pnfs: keeping the client and server 
 > > layouts in sync 
 > > 
 > > This is a long, interesting thread.  Unfortunately for you readers,
 > > I seem to have generated a long addition to it.  The executive
summary
 > > is:
 > >   I talk about ordering and deadlock.
 > >   I talk about a state generation model to reason about the
problem.
 > >   I discuss if using session sequence information is a good way
 > >     to represent the state generation model.
 > >   I conclude that using session sequence information can be
correct,
 > >     but I would prefer a cleaner protocol that makes the state
 > >     generation explicit.
 > > 
 > 
 > <snip>
 > 
 > 
 > > Suppose the server had a layout state generation number
 > > (ok, some would call that a stateid) that incremented:
 > > on each LAYOUTGET reply
 > > on each LAYOUTRETURN reply
 > > and was transmitted to the client on each CB_LAYOUTRECALL.
 > > In this model the server owns that generation number, and
 > > there is just one no matter how many clients there are.
 > > The server does not need to keep a lock held on the file
 > > while the RECALL is in-flight.  It only keeps a lock to
 > > serialize updates to the state generation number and
 > > generation of the replies or recall requests.
 > > 
 > > The function of the generation number is so the client can
 > > order the layout operations.  Suppose it did a LAYOUTGET
 > > that returned (0,1MB) at time 1.  Now suppose two things
 > > happen concurrently, another LAYOUTGET for (1MB,2MB)
 > > and a CB_LAYOUTRECALL for (0,1MB).  The server will order
 > > those two, one way or the other, and transmit them to the client
 > > that can put them back in order and process them.  Going
 > > back to my contrived example, the server might decide
 > > that the CB_LAYOUTRECALL for (0,1MB) occured at time 2,
 > > and then it replied to the LAYOUTGET for (1MB,2MB) at
 > > time 3, but passed a wider range for (0,2MB).  Even though
 > > it appears goofy that the server gave the client a layout
 > > for a range it just recalled, the sequence number makes
 > > it crystal clear what the client should do.  The client owns
 > > the layout from (0,2MB).
 > > 
 > > As an aside, the client can pass the server its last view
 > > of the state sequence number, but the server doesn't depend
 > > on that for correctness. 
 > 
 > We can conclude the server's intent by examining both replies
 > and the generation numbers in the reply, but how does the client
 > proceed in the absence of one of the replies?  For instance,
 > suppose the client receives the CB_LAYOUTRECALL response
 > with time "2".  Since time "1" was in the last reply the client
 > got from the server, it can conclude that there are no intervening
 > replies that have been mis-ordered, so it can start processing
 > this recall.  Processing this recall means sending a LAYOUTRETURN
 > for (0,1MB).  This conflicts with the reply to the LAYOUTGET that
 > is in-flight, but which the client has not yet seen.  When the
 > server receives the client's LAYOUTRETURN for (0,1MB) what should
 > it do?  I can think of at least two options, neither of which
 > seems satisfactory:
 > 
 > 1) If the client provides the latest gen number with the LAYOUT-
 > RETURN ("2"), the server can recognize that this LAYOUTRETURN
 > is not based on the latest server state.  Given this knowledge
 > the server can reject the request with some sort of new error
 > message, prompting the client to wait for a reply with the next
 > gen number to get in sync with the server.  This approach has the
 > disadvantage that it breaks our current defined semantics for
 > the LAYOUTRETURN process, and relies on the client providing
 > the latest gen number for correctness.
 > 
 > 2) It could process the layout return, and send a new reply with
 > a new gen number "4".  Now if the client is required to process
 > replies from the server is strict gen-number order, then the
 > client can sync up its state with the server by processing the
 > reply to the LAYOUTGET ("I've got 0,2MB") first, followed by the
 > reply to the LAYOURETURN ("I don't have 0,1MB").  The disadvantage
 > of this approach is that the client is left with (1,2MB) when
 > the server was trying to give it (0,2MB).  Both client and
 > server agree on the state---but its not the state that the
 > server wanted!
 > 
 > Which leads me to the simplest solution:  after issuing the
 > CB_LAYOUTRECALL, the server could reject LAYOUTGETS from the
 > client until the recall processing is finished.  Then the server
 > can grant the client (0,2MB) reliably, without the returns
 > getting in the way.  Essentially, this is the "do one thing at
 > a time" approach.
 > 
 > My point here is that while the "state sequence" view allows
 > us to unambigously reason about the correct shared state view
 > for any arbitrary interaction, the correct state may not be the
 > state that we actually desire to achieve.  I think we will always
 > need a combination of serialization to reduce the number of possible
 > interactions that we have to consider, and "state sequencing"
 > to resolve the inevitable races that occur in the interactions
 > we *do* want to consider.
 > 
 > 
 > <remaining thread snipped>
 > 
 > 

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Wed, 28 Jun 2006 23:41:44 +0300

>      resolved before proceeding to do the voluntary LAYOUTRETURN.
That's good. This means that for each {client,file} there could be at 
any time
either GETs, RETURNs, or neither. I ignore for now RETURNs and
GETs serialized in the same compound that we may want to do
eventually...

Problem 1 - the server has no way to enforce this serialization, meaning 
confused
clients can get out of sync.
>      that client will not grant any subsequent LAYOUTGETs (those not
I assume you meant _server_ will not grant...
>      in the CB_SEQUENCE) any region that overlaps that recalled.
ok, let's assume this understanding is required from the server, here 
are a few issues:

Problem 2: If there are any voluntary returns in flight to files in the 
recalled range
the client needs to wait on them to return and then decide if it needs 
to return
anything else.

Problem 3: client sent the following:
GET #1 for range 0-1
GET #2 for range 1-2
server recalls range 1-2 saying he saw GET #2
GET #2 is replied with range 0-2
In the obj-based layout the client can't split a layout by returning 
part of it so it has
to return range 0-2, yet the recalled range was 1-2.
This should fall under the voluntary return case and must be serialized with
GET #1. In general, if the client wants to return anything outside the
recalled range it should be considered voluntary.

Problem 4: client works on several recalls to the same file concurrently.
I'm not sure what the client should do here

---

From: Brent Welch <[email-masked]>
Date: Wed, 28 Jun 2006 15:35:23 -0700

>>>"Noveck, Dave" said:

I'm sorry to cloud the issue with another mechanism.  My overall
point is that "welch's state generation" and "NFSv4 stateids" and
"CB_SEQUENCE information" should be isomorphic.  They are all very
similar mechanisms proposed to solve the same problem.  Personally,
I can reason about a server-maintained state generation number.

If you say that stateids had problems with recovery, then we should
look for similar problems in the use of my state generation or
the use of session sequence information.  I believe we handle it
by revoking all the layout information.  I would agree that any
server-crash recovery requires a message exchange so the client can
resync.

In contrast, what happens when a network partition eliminates the
sessions, but the server and client remain active?  We need something
in the spec to explain how that works.

 > to deal consequences of a simple per-owner sequence number.

In a past life the state generation number was an attribute of
the file, so it was persistent.  Both/either the client or server
could crash and restart, and they could validate any cached information
quite simply.  NFSv4 stateids are per-client/per-file (at least!)
so there are a lot more of them and it is more expensive to make
them persistent.

 >      resolved before proceeding to do the voluntary LAYOUTRETURN.

I think everyone agrees on the above rule.

 >      that [the server] will not grant any subsequent LAYOUTGETs (those not

Uh, for how long?  We need more precise language for this rule.  Benny's
original thread was about this "meta operation" that spanned the
time of the server's recall to the final RETURN done by the client.
We need to define what you mean by "will not grant a LAYOUTGET"
(NFS4_RETRY ? NFS4_LAYOUTDENIED?), and we need to bound the time
during which the server doesn't grant a layout that it has just
recalled.

I will refer again to my example where the client does concurrent
RETURN and GET for the same range.  I don't see how the session
sequence numbers solve that problem.  The state generation number
doesn't make any rules about what the server does, but it makes
the server's serialization explicit so the client knows how to respond.

If you are going to impose a rule that the client has to issue all
GET and RETURN operations over the same session slot, then you'll
have to open up an wider interface to the RPC transport layer.
Again, in a past life I implemented RPC sessions and they were
hidden below the RPC transport interface.  I didn't have any higher
level protocols that knew about sessions.

 > motivation.  Why is it that a simpler approach won't work or is

heh - I view a single state generation number as simpler than
a concatenation of session sequence numbers.  I think you view
it as more complicated because there is an implication of
persistence during recovery.

Again, I'm not hung up on having a state generation number as
the implementation, but I feel pretty strongly that it is a good
model for what happens in the distributed system.  If you can
map the CB_SEQUENCE implementation to an abstract model of
a state generation number, then I'd be confident it was correct
and complete.

No matter what you do, we need a concrete rule for what happens
during the interval between the RECALL and the subsequence RETURNS
and GETs done by clients.

 > Subject: Re: [nfsv4] pnfs: keeping the client and server layouts in =
 > sync
 > 
 > 
 > 
 >  > > This is a long, interesting thread.  Unfortunately for you =
 > readers,

www.panasas.com
[email-masked]