Subject: [nfsv4] Issue in LAYOUTRETURN

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 14 Nov 2007 15:55:28 -0500

In editing this to deal with another issue, I noted the following:

     The server MAY require that the principal, security flavor, and
     applicable, the GSS mechanism, combination that acquired the
     layout also be the one to issue LAYOUTRETURN. This might not be
     possible if credentials for the principal are no longer
     available. The server MAY allow the machine credential or SSV
     credential (see <xref target="OP_EXCHANGE_ID" />) to issue
     DELEGRETURN.

If the client does the former without doing the latter, it is 
going to have a heck of a time doing LAYOUTRETURN_FSID or
LAYOUTRETURN_ALL.  Do we want to say something about this.

Depending on your interpretation of:

      However,
      if it is a subset, the recall is not complete until the full
      recalled scope has been returned.  Recalled scope refers to the
      byte range in the case of LAYOUTRETURN4_FILE, use of
      LAYOUTRETURN4_FSID, or the use of LAYOUTRETURN4_ALL.

it may not be possible to respond to LAYOUTRECALL of ALL, if
the requirement is that the client actually do a completing 
LAYOUTRETURN4_ALL, rather than simply returning each and 
every layout he holds.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Wed, 14 Nov 2007 19:36:20 -0800 (PST)

>      credential (see <xref target="OP_EXCHANGE_ID" />) to issue
       ^^^^^^^^^^^^

Is DELEGRETURN really meant here, or is it LAYOUTRETURN?

> LAYOUTRETURN_ALL.  Do we want to say something about this.

Makes no difference to me. I'm sure some clients will fail to take
advantage of mechanism no matter what we write down.

> LAYOUTRETURN4_ALL, rather than simply returning each and 

Beats me. This _FILE and _ALL stuff looks quarter baked to me.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 15 Nov 2007 15:58:18 +0200

>>      credential (see <xref target="OP_EXCHANGE_ID" />) to issue

It looks like a typo to me.  Anyhow I don't understand the motivation
for such requirement for LAYOUTRETURN.  I'd be happier if the server
will be required to allow the machine or SSV creds to issue LAYOUTRETURN
and probably also DELEGRETURN as the layout and the delegation may
used by the client host for any user.  They use OPEN and ACCESS
to authenticate and authorize each user but they are not required to
get a LAYOUT and/or delegation on each user's behalf.

> Beats me. This _FILE and _ALL stuff looks quarter baked to me.

FWIW, the way I implemented the linux client is that for RECALL FSID or
ALL it first returns all layouts in the recalled scope and then it
send the corresponding RETURN_FSID or RETURN_ALL that completes
the layoutrecall process.  At this point the client knows of no
file it has a layout for with the recalled layout_type, iomode, and
fsid in the RECALL_FSID case.  It is possible though that the client
threw away such a layout without previously returning it and the
"wildcard" return will clean that up also on the server side.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Thu, 15 Nov 2007 06:07:32 -0800 (PST)

> It looks like a typo to me.  Anyhow I don't understand the motivation

Yep. Fixed.

> for such requirement for LAYOUTRETURN.  I'd be happier if the server

Andy a similar question which I responded to.

> will be required to allow the machine or SSV creds to issue

The protocol does allow the client to ask the server to allow
this.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 15 Nov 2007 15:18:40 -0600

> LAYOUTRETURN4_ALL, rather than simply returning each and

I would prefer that we view the "FSID" and "ALL" variants of
the recall and return processing as a shorthand reference to
the individual layout detail.  Therefore, if the server
does a recall of "ALL", then the client can return the
individual layouts or it can to the return of "ALL".  Conversely,
if the server recalls an individual layout, the client
can do a return of "ALL" and have it satisfy the recall
processing.

Half-baked??.  It seems valuable to a client in the event of
a client dealing with an unmount of a filesystem or shutdown
of the system.  Of course, we don't have a corresponding
CLOSE("ALL) or DELEGRETURN("ALL) so maybe that would argue
that without the symmetry the feature isn't needed and may
lead to lazy/broken client implementations.

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 15 Nov 2007 16:25:42 -0500

> lead to lazy/broken client implementations.

Delegations aren't necessarily tied to user open state. There is value
for a client to keep holding delegations to often-used files (for
instance shared libraries) even if nobody happens to have a file open at
the moment. For such cases, being able to issue a single return at
umount is valuable; particularly so for read delegations, where there is
no dirty state to worry about.

Trond

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 15 Nov 2007 15:29:28 -0600

> no dirty state to worry about.

One could argue that a SETCLIENTID or EXCHANGE_ID is a method of
exacting that DELEGRETURN.  :-)

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 15 Nov 2007 16:32:00 -0500

> exacting that DELEGRETURN.  :-)

Only if you're prepared to set up one clientid per filesystem you
mount. :-)

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 15 Nov 2007 22:23:18 -0500

And you're prepared to close all files, release all byte-range locks,
at the same time.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 15, 2007 4:32 PM
To: Spencer Shepler
Cc: Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] Issue in LAYOUTRETURN

> exacting that DELEGRETURN.  :-)

Only if you're prepared to set up one clientid per filesystem you mount.
:-)

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Nov 2007 11:02:29 -0500

> individual layouts or it can to the return of "ALL". 

I would prefer that interpretation as well.

However, it conflicts, at least in spirit, with the following
from LAYOUTRECALL:

   However, the last LAYOUTRETURN in a sequence of
   returns, MUST specify the full range being recalled (see
   Section 12.5.4.1 for details).

I think we should rethink this for draft-18.  The reasoning
in Section 12.5.4.1 doesn't convince me and the working group
should come to a consensus on this issue.

> of the system. 

I think we need to separate three different questions.

1.  How baked, i.e. clearly specified without conflicts with 
    other pieces of the spec, this is.

2.  How good this will taste, i.e. convenient and useful it
    will be, when it is fully baked.

3.  How necessary it is, i.e. whether without it there will be
    severe difficulty in doing pNFS implementations.

I think we are at the point where things should not go into the
protocol where they are not OK, with respect to 1+2.

Soon, we will need to raise that bar and insist that changes not 
go in unless that are OK with regard to 1+3.  If the bar remains
at 1+2, we will never close.  With this group of people, we will
always be able to improve the protocol, and come up with things
that, when well-baked, will taste fine, and we will be able to 
get them well-baked, through group discussion.  But the result 
will be the finest protocol that never actually got finished.  
None of us would like that.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Thursday, November 15, 2007 4:19 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Issue in LAYOUTRETURN

Spencer

---

From: [email-masked]
Date: Fri, 16 Nov 2007 13:05:16 -0500

Dave, 

> should come to a consensus on this issue.

If a client can free up the layouts at different times, it can
return the individual layouts earlier than doing the ALL return.
Doing ALL as the final return helps deal with possible state
mismatches (different views on which layouts are held) between
client and server.

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Nov 2007 17:37:20 -0500

I have trouble believing that even with this approach to matching of
LAYOUTRETURN and CB_RECALLLAYOUT we can survive a situation in which
the client and server have divergent views of what layouts are held.
In particular, the addition of stateids to this logic, doesn't seem
very consistent with a protocol intended to have the client and 
server allow such differences to develop and somehow accommodate
them.

I do a LAYOUTGET and I get a stateid, which I then use in a subsequent
LAYOUTGET, but I send it with seqid zero to accommodate the possiblity
of parallel LAYOUTGET's.  The seqids on the stateids coming back
are not going to coherent if the server has not maintained information,
including stateids, on the layouts he thinks (or rather know) I have.
Similarly with voluntary LAYOUTRETURN's.  

If we want to design the protocol so that such diveregences in layout
knowledge can be accommodated, I think we need some careful review,
to make sure that the protocol as a whole can work in the presence 
of such divergences.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, November 16, 2007 1:05 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: RE: [nfsv4] Issue in LAYOUTRETURN

Dave, 

> > I would prefer that we view the "FSID" and "ALL" variants of the 
> > recall and return processing as a shorthand reference to the 
> > individual layout detail.  Therefore, if the server does a recall of

> > "ALL", then the client can return the individual layouts or it can 
> > to the return of "ALL".
> 
> However, it conflicts, at least in spirit, with the following from 
> LAYOUTRECALL:
> 
> I think we should rethink this for draft-18.  The reasoning in Section

> 12.5.4.1 doesn't convince me and the working group should come to a 
> consensus on this issue.

If a client can free up the layouts at different times, it can return
the individual layouts earlier than doing the ALL return.
Doing ALL as the final return helps deal with possible state mismatches
(different views on which layouts are held) between client and server.

Thanks,
--David

---

From: Spencer Shepler <[email-masked]>
Date: Sat, 17 Nov 2007 23:19:29 -0600

> to make sure that the protocol as a whole can work in the presence

To support Dave's observations in differing language.  Byte range
file locking isn't expected to diverge and layouts should be treated
no differently.  If we are planning for failure, that is what we will
receive and deserve.

Spencer

> mismatches

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 18 Nov 2007 10:22:12 -0500

There are a couple of differences between layouts and byte-range locks
that I think are important here:

     Layouts are recallable, leading to a worry that a server, by
     recalling some set of byte ranges, may impose on the client a
     complicated set of discontiguous ranges, which it might be
     inconvenient for the client to remember.

     The server, unlike the case of locks, may respond to a request 
     with something different than the client requested, leading to
     more complex information for the client to save.

     Layouts, at least potentially, are going to be used more 
     intensively than byte-range locks, making expansion of the
     size of data involved, for both the client and the server,
     more of an issue.

The current spec has some places where it tries to address these
issues by suggesting as David indicates that we try to accommodate
the inconsistencies that would arise by simply allowing the client
and the server to each forget layout information at will.  I have
some worries about whether that can be made to work in all cases, 
even without layout stateids.  But it seems to me that with layout 
stateids, it gets harder.

I think we need to decide what we want to do about this issue and
the choices I know about are:

     Make this ability to independently forget layout information
     the centerpiece of the protocol, and drop layout stateids.

     Figure out how to have that ability work together with 
     stateids, convince ourselves we have done it, and clearly
     explain in the spec, how that works.

     Go to a strict agreement model as Spencer is suggesting and
     explain the protocol from that standpoint.  

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Sunday, November 18, 2007 12:19 AM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] Issue in LAYOUTRETURN

On Nov 16, 2007, at 4:37 PM, Noveck, Dave wrote:

> very consistent with a protocol intended to have the client and server

> allow such differences to develop and somehow accommodate them.
>

> of parallel LAYOUTGET's.  The seqids on the stateids coming back are 
> not going to coherent if the server has not maintained information, 
> knowledge can be accommodated, I think we need some careful review, to

> make sure that the protocol as a whole can work in the presence of 
> such divergences.

To support Dave's observations in differing language.  Byte range file
locking isn't expected to diverge and layouts should be treated no
differently.  If we are planning for failure, that is what we will
receive and deserve.

Spencer

> mismatches (different views on which layouts are held) between client 
> and server.
>
> Thanks,
> --David
> David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,

> Hopkinton, MA  01748

---

From: [email-masked]
Date: Mon, 19 Nov 2007 14:29:58 -0500

Dave,

EMC has implementation experience from MPFS/FMP which suggests

>      explain in the spec, how that works.

has robustness and flexibility gains for client implementations.

In the former case, when the server issues a return for a set of
layouts and/or ranges, the client has to return *exactly* what
the server thinks it has.  Taken to its extreme limit, it can
yield situations in which an out-of-sync client has no chance
of working further with the server, and it imposes server
communication costs on client decisions to drop otherwise-clean
layout state.

In the latter case, when the server issues a return for a set of
layouts and/or ranges, the client has to get to a state in which
it no longer has anything that needs to be returned and tell the
server that it has reached that state.  This allows some out-of-
sync situations to be resolved cleanly, and allows some client
optimizations to drop no-longer-needed state without having to talk
to the server.  

The server can't tell these two situations apart for a client
that issues returns based on "I don't have any of those".  Think
about the responses to the Old Maid "Give me all your 8's"
request - whether or not the recipient had any 8's beforehand,
she will have no 8's when her response is complete, and the
latter is what the server actually cares about (no outstanding
layout or range covered by issued recall).  Of course, the client
*must* absolutely respect the stateid protocol as part of this -
getting confused about the stateid is generally fatal to continued
use of the state protected by the stateid.

One new wrinkle here is that client discard of clean state need
not affect the stateid, and that probably falls into your category
of "convince ourselves we have done it, and clearly explain in
the spec, how that works."  The other new wrinkle is that if a
client gets a recall for a layout that it has discarded, it needs
to have a clean way to tell the server "I don't have that layout",
and I believe that's already there as an error return to the callback.

Thanks,
--David

> From: Noveck, Dave [mailto:[email-masked]] 
> Sent: Sunday, November 18, 2007 10:22 AM
> To: Spencer Shepler
> Cc: Black, David; [email-masked]

---

From: Benny Halevy <[email-masked]>
Date: Mon, 19 Nov 2007 21:38:52 +0200

>      the centerpiece of the protocol, and drop layout stateids.

It's not exactly the way it is described right now.
The client and server are allowed to simplify their
layout state but only in one direction that results in
the server assuming the client has more layouts than
it thinks it has.

This is encapsulated here:
   o  It may be useful for clients to be able to discard layout
      information without calling LAYOUTRETURN.  If conflicts that
      require callbacks are very rare, and a server can use a multi-file
      callback to recover per-client resources (e.g., via a FSID recall,
      or a multi-file recall within a single compound), the result may
      be significantly less client-server pNFS traffic.

   o  It may be similarly useful for servers to maintain information
      about what ranges are held by a client on a coarse-grained basis,
      leading to the server's layout ranges being beyond those actually
      held by the client.  In the extreme, a server could manage
      conflicts on a per-file basis, only issuing whole-file callbacks
      even though clients may request and be granted sub-file ranges.

   o  In order to avoid errors, it is vital that a client not assign
      itself layout permissions beyond what the server has granted and
      that the server not forget layout permissions that have been
      granted.  On the other hand, if a server believes that a client
      holds a layout that the client does not know about, it is useful
      for the client to cleanly indicate completion of the requested
      recall either by issuing a LAYOUTRETURN for the entire requested
      range or by returning an NFS4ERR_NOMATCHING_LAYOUT error to the
      CB_LAYOUTRECALL.

   Thus, in light of the above, it is useful for a server to be able to
   issue callbacks for layout ranges it has not granted to a client, and
   for a client to return ranges it does not hold.  A pNFS client MUST
   always return layouts that comprise the full range specified by the
   recall.  Note, the full recalled layout range need not be returned as
   part of a single operation, but may be returned in portions.  This
   allows the client to stage the flushing of dirty data, layout
   commits, and returns.  Also, it indicates to the metadata server that
   the client is making progress.
...
   In order to ensure client/server convergence with regard to layout
   state, the final LAYOUTRETURN operation in a sequence of LAYOUTRETURN
   operations for a particular recall, MUST specify the entire range
   being recalled, ...

>      explain the protocol from that standpoint.  

We've been advocating that for a long time.
I think that having layout stateids allows us to implement a strict model
though it will probably require the removal of RETURN_{FSID,ALL}
from the protocol as the layout stateid is per-file.
I guess that Mike Eisler would also want to get rid of RECALL_{FSID,ALL}
in this occasion, but I'm not sure if this will be required.  That depends
on the semantics we define for the layout stateid given in the CB_LAYOUTRECALL
args.  If it has strict ordering semantics with respect to outstanding
LAYOUTGETs and LAYOUTRETURNs then it will be a problem to correlate
that stateid with any of the individual file layout's being recalled.
That said, I'm not how this works together with the referring_call_list
on CB_SEQUENCE.  Ideally, I'll be very happy if the layout stateid on
CB_LAYOUTRECALL can replace the sessions callback based mechanism
for resolving the potential race as the former is simpler to implement.

Benny

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 19 Nov 2007 19:57:32 -0600

>      more complex information for the client to save.

In both of these cases the client is not bound to use the layout
information.  In the first case, it may return layout ranges to
ease the implementation burden or to ease issues related to
performance.  It may also return the layouts and attempt to
acquire new ones.  In the second case, the client is not bound
by the server either given that the client always has the option
to read/write via the meta-data server.

The server implementations will likely adapt their behaviors
over time to allow for reasonable performance and interoperability
with client capabilities.

>      size of data involved, for both the client and the server,

This view seems to presume complex or discontiguous layouts.
Is this likely to be the case and if we don't know are we
comfortable in a design that make this assumption.  I don't have
a strong opinion here other than my original implicit assertion
about maintaining a shared common view of layout state
between client and server.

> stateids, it gets harder.

I agree.  And this seems to be the first principal that we are
revisiting at this point.  In general, for all layouts, is the
design goal to allow the client (in this discussion) the ability
to "forget" about layout ranges.  By this I assume that this
forgetfulness is in the form of not doing a LAYOUTRETURN and then
being able to respond to a LAYOUTRECALL in a way to inform the
server that the layout is not currently held.

>      explain the protocol from that standpoint.

To assist in making the decision about direction, it would be helpful
to have concrete use cases of when the server will (either in
implementation today or future plans) recall layouts.  And then
use cases for the client in its need for optimization of LAYOUTRETURNs
in that it is anticipated to be too much overhead for client capture
of the data, over the wire interaction or the combination of the two.

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 19 Nov 2007 20:05:19 -0600

> communication costs on client decisions to drop otherwise-clean

Can you add some words or experience to the case where the client
is out-of-sync with the server?  Is this by choice or by
implementation or protocol error?

> optimizations to drop no-longer-needed state without having to talk

In the implementation you mention, does the client associate
layout information with cached pages and when those pages are
flushed the layout information is "dropped" and thus answering
my question above or is there some other condition that occurs, etc.

> use of the state protected by the stateid.

 From the point of view of a protocol that requires the client and  
server
to share the layout state, the only time the client would need to
know exactly what it has is in the case where no recall of layout is
being done.  In the servicing of the recall, the client can just pretend
it has the layouts for which the server is asking.

To support the forgetful client, the layoutreturn would need to
allow for a "0->max-range" layoutreturn and the stateid transition
would need to occur for that to signify that the server has
processed the request (even if no layouts were being held).

> the spec, how that works."

Can you expand on what you intend by "client discard of clean state"?
What scenario does this describe?

Spencer

> The other new wrinkle is that if a

---

From: [email-masked]
Date: Wed, 21 Nov 2007 18:52:12 -0500

Spencer,

I should let Jason elaborate further, as he's closer to these

> implementation or protocol error?

It can be by implementation choice, and can help with some protocol
errors.  We initially did it as an implementation choice.

> > This allows some out-of-

Yes, that is an example, and it avoids having to return a layout
when dropping clean pages from cache.

server
> it has the layouts for which the server is asking.

Right - the ability of the client to drop state assumes that
if the server needs something, it'll send a recall.

> processed the request (even if no layouts were being held).

Probably also a good idea, if the client wants to do the server
a favor, enabling the server to not send a recall.

> What scenario does this describe?

The discard of a clean page and associated layout state from the cache,
as you summarized above.

Thanks,
--David

> Sent: Monday, November 19, 2007 9:05 PM
> To: NFSv4

---

From: [email-masked]
Date: Mon, 26 Nov 2007 14:28:17 -0500

The most relevant cases that I am familiar with are the following:

Case 1
The client issues a LAYOUTGET operation and the server returns a LAYOUT
comprised of many extents.  The client while processing the individual
extents and putting them into client data structures runs out of memory.
The cleanest behavior in this situation is to silently drop the
remaining extents that comprise the layout.

Case 2
A similar case happens under memory pressure.  In our implementation of
FMP, the client implementation releases read-only layouts when notified
by the memory manager to shrink its cache.  In such a situation it is
advantageous to immediately drop the resources.  In the short term,
acquiring new resources to send an RPC would be counter productive.

I think that the protocol should allow for both behaviors.

-Jason

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, November 21, 2007 6:52 PM
To: [email-masked]; [email-masked]
Subject: RE: [nfsv4] Issue in LAYOUTRETURN

Spencer,

server

Thanks,
--David