Subject: RE: [nfsv4] pnfs: keeping the client and server layouts in sync

From: [email-masked]
Date: Tue, 27 Jun 2006 22:02:05 -0400

Brent---
This is a good analysis of the problem, but I'm not sure I
agree that the "state sequence" view will completely address
all state synchronization problems by itself.  Even with
such a mechanism, I think you will still need to introduce
some serialization at the server.  See my comments
on your example below...  

-Steve

> -----Original Message-----
> From: Brent Welch [mailto:[email-masked]] 
> Sent: Tuesday, June 27, 2006 6:03 PM
> To: Benny Halevy
> Cc: Talpey, Thomas; [email-masked]
> Subject: Re: [nfsv4] pnfs: keeping the client and server 
> layouts in sync 
> 
> This is a long, interesting thread.  Unfortunately for you readers,
> I seem to have generated a long addition to it.  The executive summary
> is:
>   I talk about ordering and deadlock.
>   I talk about a state generation model to reason about the problem.
>   I discuss if using session sequence information is a good way
>     to represent the state generation model.
>   I conclude that using session sequence information can be correct,
>     but I would prefer a cleaner protocol that makes the state
>     generation explicit.
> 

<snip>

> Suppose the server had a layout state generation number
> (ok, some would call that a stateid) that incremented:
> on each LAYOUTGET reply
> on each LAYOUTRETURN reply
> and was transmitted to the client on each CB_LAYOUTRECALL.
> In this model the server owns that generation number, and
> there is just one no matter how many clients there are.
> The server does not need to keep a lock held on the file
> while the RECALL is in-flight.  It only keeps a lock to
> serialize updates to the state generation number and
> generation of the replies or recall requests.
> 
> The function of the generation number is so the client can
> order the layout operations.  Suppose it did a LAYOUTGET
> that returned (0,1MB) at time 1.  Now suppose two things
> happen concurrently, another LAYOUTGET for (1MB,2MB)
> and a CB_LAYOUTRECALL for (0,1MB).  The server will order
> those two, one way or the other, and transmit them to the client
> that can put them back in order and process them.  Going
> back to my contrived example, the server might decide
> that the CB_LAYOUTRECALL for (0,1MB) occured at time 2,
> and then it replied to the LAYOUTGET for (1MB,2MB) at
> time 3, but passed a wider range for (0,2MB).  Even though
> it appears goofy that the server gave the client a layout
> for a range it just recalled, the sequence number makes
> it crystal clear what the client should do.  The client owns
> the layout from (0,2MB).
> 
> As an aside, the client can pass the server its last view
> of the state sequence number, but the server doesn't depend
> on that for correctness. 

We can conclude the server's intent by examining both replies
and the generation numbers in the reply, but how does the client
proceed in the absence of one of the replies?  For instance,
suppose the client receives the CB_LAYOUTRECALL response
with time "2".  Since time "1" was in the last reply the client
got from the server, it can conclude that there are no intervening
replies that have been mis-ordered, so it can start processing
this recall.  Processing this recall means sending a LAYOUTRETURN
for (0,1MB).  This conflicts with the reply to the LAYOUTGET that
is in-flight, but which the client has not yet seen.  When the
server receives the client's LAYOUTRETURN for (0,1MB) what should
it do?  I can think of at least two options, neither of which
seems satisfactory:

1) If the client provides the latest gen number with the LAYOUT-
RETURN ("2"), the server can recognize that this LAYOUTRETURN
is not based on the latest server state.  Given this knowledge
the server can reject the request with some sort of new error
message, prompting the client to wait for a reply with the next
gen number to get in sync with the server.  This approach has the
disadvantage that it breaks our current defined semantics for
the LAYOUTRETURN process, and relies on the client providing
the latest gen number for correctness.

2) It could process the layout return, and send a new reply with
a new gen number "4".  Now if the client is required to process
replies from the server is strict gen-number order, then the
client can sync up its state with the server by processing the
reply to the LAYOUTGET ("I've got 0,2MB") first, followed by the
reply to the LAYOURETURN ("I don't have 0,1MB").  The disadvantage
of this approach is that the client is left with (1,2MB) when
the server was trying to give it (0,2MB).  Both client and
server agree on the state---but its not the state that the
server wanted!

Which leads me to the simplest solution:  after issuing the
CB_LAYOUTRECALL, the server could reject LAYOUTGETS from the
client until the recall processing is finished.  Then the server
can grant the client (0,2MB) reliably, without the returns
getting in the way.  Essentially, this is the "do one thing at
a time" approach.

My point here is that while the "state sequence" view allows
us to unambigously reason about the correct shared state view
for any arbitrary interaction, the correct state may not be the
state that we actually desire to achieve.  I think we will always
need a combination of serialization to reduce the number of possible
interactions that we have to consider, and "state sequencing"
to resolve the inevitable races that occur in the interactions
we *do* want to consider.

<remaining thread snipped>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Todd Pisek <[email-masked]>
Date: Wed, 28 Jun 2006 09:48:38 -0500

WRT the use of stateid's for data server validation of I/O
requests, is the assumption that all the data servers in
all the layout segments of a given layout associated with
an open instance use the same stateid?

Poor attempt at ascii graphics:

    open instance A     open instance B
    stateid A           stateid B
         \               /
          \             /
           \           /
          layout segments
           /      |     \
          /       |      \
         /        |       \
data server 1    ds2 .... dsN

write using instance A passes state id A
to every data server.

write using instance B passes state id B
to every data server.

--- Todd

---

From: Brent Welch <[email-masked]>
Date: Wed, 28 Jun 2006 09:21:22 -0700

[email-masked] said:
> I think you will still need to introduce some serialization at the server.

The sequence number represents the serialization that has occurred.

Benny points out that it is the ordering between LAYOUTGET and
LAYOUTRETURN that is most important. The RECALL is a kind of
hint to the client that it is supposed to do something.  (Of
course, if you ignore the hints, the server will fence you.)
So, let's add LAYOUTRETURN to these examples:

LAYOUTGET(0,1MB) at time 1
CB_LAYOUTRECALL(0,1MB) at time 2
LAYOUTRETURN(0,1MB) at time X
LAYOUTGET(1MB,2MB) at time Y, but returning (0,2MB)

if (X < Y), then the server processed the return before the get,
and the client owns (0,2MB).  This scenario is that the server
wanted to do a recall, got the return, but then "changed its mind"
and decided to give the layout back to the client.

If (Y < X), then the server processed the get first and then
the return, so the client owns (1MB,2MB).  This matches our
expectations.

To support this, the client needs to remember the state sequence
number associated with its layouts.  (By analogy, I think with
the session sequence information you need to remember all of that.)
This lets the clients ignore stale responses.
I think this holds water even if layout ranges are subdivided.

>>>[email-masked] said:
 > 
 > Brent---

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Garth Goodson <[email-masked]>
Date: Wed, 28 Jun 2006 15:50:51 -0700

> an open instance use the same stateid?

Yes.  Whichever stateids could be used on the metadata server (or in 
regular V4) must be accepted on the data servers for those ios..

-Garth