Subject: [nfsv4] re-acquiring delegations

From: Mike Eisler <[email-masked]>
Date: Tue, 14 Mar 2006 11:25:07 -0800 (PST)

In NFSv4.0 a client can acquire a delegation and
have it recalled whenever there is contention on it,
or perhaps resource pressure on the NFSv4 server.

The only way to re-obtain a delegation is for the
client to re-open the file.

At Connectathon two weeks ago, someone reminded us
that this was something we wanted to address but
currently it is not covered in the NFSv4.1 i-d
or any associated i-ds.  Of course is this is a
significant last minute change to NFSv4.1.  I know
Brian Pawlowski, our co-chair, has been worried about
this issue, so perhaps there is a co-chair's privilege
rule that Spencer neglected to tell us about.

I've written down some thoughts. I haven't thought
about whether any it applies to layouts.

There are questions to be answered:
- do we want to do something in NFSv4.1 about this?
- what is the problem?
- what are the requirements?
- what are the solutions?

I'll leave it to the WG to reach consensus combined
with the rule of the co-chairs to answer the first
question. The rest I propose answers for here.

The problem statement is: how should NFSv4 clients
optimally re-acquire delegations?

What are the requirements?

R1. If a client desires to cache file, when conditions
  improve to allow re-delegation, the client learns
  of those conditions within N seconds.

Depending on what N is, I can see multiple solutions.

N is on the order of minutes or more:

S1. The client can use the existing NFSv4.0 facility
  to OPEN the file again, with the same open_owner as
  before, and either get a delegation or not. I.e.,
  clients can poll for delegation availability.

	Advantage: no changes to NFSv4.x protocol
			needed.

	Disadvantages:

		- If the client has lots of files
		  and/or there are lots of clients,
		  the NFSv4 server can be hammered
		  with requests for re-acquiring
		  delegations. This doesn't seem
		  scalable.

N is on the order of seconds or more:

S2. When the client returns the delegation, or when
  a client opens a file and fails to get a delegation,
  or when a client attempts a GET_DIR_DELEG operation,
  it indicates to the server it wants the delegation
  when it is available. This is done via new
  operation, WANT_DELEG, which takes the current file
  handle as the first argument, and the delegation
  type, SHARED, EXCLUSIVE, EITHER as the other (the
  directory notification stuff is something I haven't
  thought much about, but I assume it can be handled
  inside WANT_DELEG or perhaps a WANT_NOTIFY). If the
  client already has a delegation, the server is free
  to ignore WANT_DELEG. Note that this offers the
  possibility of the server granting atomic upgrade
  from SHARED to EXCLUSIVE, but by no means is a
  server required to do so. WANT_DELEG can fail. If it
  fails with new NFSv4 error NFS4ERR_TOO_MANY_WANTS,
  the client knows the server won't keep track of
  this delegation want.

  When it becomes available, the server, via a new
  CB_PUSH_DELEG operation, gives the delegation to
  the client. The client will have the opportunity to
  accept or reject. In either case, the server MUST
  no longer track the client's interest in future
  delegations on the file.

  (I suspect that without sessions, there are
  race conditions with CB_PUSH_DELEG.  I suggest
  making CB_PUSH_DELEG and therefore WANT_NOTIFY,
  depend on sessions. I was also torn about pushing
  a delegation, versus pushing an indication that a
  delegation is available.  I realize this contradicts
  my negative views on call backs in byte range file
  locking -- in my defense, it is a much bigger deal
  for a client to be confused about whether it has
  a byte range lock versus confused about whether it
  has a delegation; the latter is recallable.)

  One of the reasons why a delegation can be recalled
  is resoure limits on the server. And one of the
  reasons why delegations might not be re-acquirable
  is resource limits on the server. Thus, it would
  nice if the server pushed the delegations the client
  was interested in.

  To this end, WANT_DELEG takes a third argument with
  one of 4 values.

     WANT_DELEG_PRI_HI = 0,
     WANT_DELEG_PRI_LO = 1,
     WANT_DELEG_PRI_UNSPEC = 2,
     WANT_DELEG_PRI_DONT_WANT = 3
  
  If the priority is HI, this tells the server that a
  delegation on this file is client's highest priority
  among a set of delegations. If the priority is LO,
  this tells the server that a delegation on this file
  is the client's lowest priority. If it is UNSPEC,
  this tells the server the client has indicated no
  preference. In terms of implementation, the the
  server might maintain a priority queue, moving file
  handles to the front or back of the queue depending
  on whether WANT_DELEG has a 3rd argument of HI or
  LO. If DONT_WANT is set, this means the client
  is not interested in a delegation for the file.
  A server is not required to pay attention to the
  priority orders from the client, but it SHOULD
  do so.

	Advantage: 
                - more scalable.

                - the client gets to re-acquire the
                  delegations it is most interested in

                - the intelligence is client-centric
                  per the tradition of NFS

	Disadvantages:
		- more state required by server in
		  addition to open state

There is actually a continuum of solutions between
S1 and S2, but they seemed like dead ends in the sense
that they introduced new operations that S2 wouldn't
need. S2 seemed to be where we wanted to end up.

   -mre

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 14 Mar 2006 13:32:48 -0600

>From a personal perspective, this issue should be
addressed in 4.1 given that we are adding directory
delegations and I believe the next order issue will
be a client that needs to do the simple task of
obtaining a delegation for one previously held.

As WG co-chair, I am also in favor of this given
the quality of specification issue; would like
to have something that has a high probability of
sticking.  Of course, this statement will likely
come back to haunt me.

However, I reserve the right to change the above
opinions if the eventual solution proves to be
too complex with respect to "schedule".

Spencer

---

From: Carl Burnett <[email-masked]>
Date: Wed, 15 Mar 2006 20:09:09 -0600

Just a note that in my review of the early directory delegation drafts I 
had proposed that it address the general notion of requesting a delegation 
for an already opened file that lost its delegation due to a recall.

That suggestion was just for this case of re-obtaining a delegation after 
a recall.

Carl

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

---

From: Mike Eisler <[email-masked]>
Date: Mon, 20 Mar 2006 14:45:46 -0800 (PST)

> That suggestion was just for this case of re-obtaining a delegation after 

Thanks Carl.

Do you think my "S2" proposal is overkill, or would it suffice to
have a separate client driven operation to allow a client to
request a delegation on an already OPENed file?

(My S1 suggested just using OPEN to do this, but I the problem that has
over a separate GET_DELEG_FILE operation is that if the OPEN succeeds
but fails to get an delegation, the client has to follow up with a CLOSE).

Thus far in this discussion, only Carl and Spencer have voiced
an opinion that we need to do something. I'd  be happy to write up
a proposal for "S2" or anything else, but I'm not seeing a rough
consensus. So it appears there's no overall desire to address this.

I will resolve the issue in the issues tracker (
http://www.nfsv4-editor.org/cgi-bin/roundup/nfsv4/issue2 )
to say:

  No WG consensus to solve this issue

if I don't hear otherwise by Tuesday, March 21, 5pm pacific time.

NFSv4.1 will recommend clients use