Subject: RE: [nfsv4] Directory delegations and notifications

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 19 Apr 2005 09:57:14 -0400

Let me first address the issue of write directory delegations.  It 
seems to me that this just adds too much complexity and that given
the problems that this would raise, it just isn't ready for v4.1.
Let me cite a few issues:

Allowing creation of files in write-delegated directories raises
the issue of fileid assignment.  If you create a file, it presumably
has to be statable at that point, so that it needs to have a fileid
assigned, but if the client makes that choice, then how can it choose
a value that will be acceptable to the server and capable of being 
maintained as the fileid when the directory delegation is returned?
This is not an insoluble problem but it is not trivially solvable 
either.

The validity of filenames within a directory has significant elements
of server choice.  The server may reject certain UCS characters 
(e.g. those beyond 0xffff), and may reject certain reserved names
(e.g. ".", "..") and the server has a fair amount of freedom in this
regard.  The server's rules would have to be externalized so that the
delegated client could do the same thing that the server would do.
There's also the issue of possible normalization rules that would
add to the complications.

There are significant issues due to the fact that server semantics is
unspecified in other respects.  For example, servers may, but need
not, reject REMOVE and RENAME requests for open files.  If you have
directory write delegation then it is the client who will be processing
such requests.  You either have to specify the semantics so that 
everybody agrees what the proper handling is (and it is hard to get
agreement which is precisely why this was left unspecified) or have
the server specify his choices (with all possibilities accommodated)
so that the client can do what the server would have done.

Server reboot and reclaim issue are complicated.  A read directory
delegation can simply be considered to have been lost on reboot
but with a write directory delegation you may have pending changes
and so you have to be able to reclaim it after reboot.  But if you
can reclaim it within the grace period, you must reject requests
that might be invalidated should there be a directory write delegation.
So it looks like LOOKUP, READDIR, RENAME, REMOVE, CREATE, GETATTR,
LINK at least need to be able to return GRACE.  So clients have to
be able to do their reclaims without any of those operations which 
is possible but may pose difficulties for some clients.  Note here
that clients which do not have any directory delegations will still
get the error.  Also v4.0 clients must be dealt with and they will
have to get DELAY instead of GRACE.  I think this area needs a lot
more thought and discussion.

All of these issues can be dealt with but not in a v4.1 timeframe 
unless that timeframe is very much extended.  I think we are better 
off with read directory delegation in v4.1 and write directory 
delegation later when we have a consensus on how to address all 
the issues.

As far as notifications go, I think Saadia is right to point out
the performance issues that we may face unless we do something
about the issue of very large read-mostly directories.  I would
hate to be in a situation in which we did the work for directory
read delegation and wind up with a situation in which they were
a win for a restricted set of workloads and had to wait for broader
performance improvement in v4.2.  If we can address the case of very 
large read-mostly directories without making v4.1 too complicated, 
then that's what we should do.

So let's explore how we might address this issue without having
notifications.  The following is an idea that I may have mentioned
to some of you.  Since I think I had a drink in my hand at the time,
perhaps it wasn't given the most coherent formulation.  Anyway, the
attempt is to do the minimum that will address the VLRMD problem
(very large ready-mostly directories).

So the problem is that even if a modest rate of change, the need to
reread an entire large directory at each change/recall-delegation/
re-get-delegation cycle.  So how can we avoid this overhead?  Suppose,
when you got a directory delegation you got a opaque cookie that 
designated the current state of that directory including file attributes.
The validity of that cookie would only be guaranteed for the life of
the delegation, but, the server may keep it valid for longer.  The
only requirement on the server is that even if it the cookie is no
longer valid, it must be able to recognize that it is out-of-date.  Like
filehandles, they never can be reused.												

So now suppose a delegation is recalled, returned, and regotten.  The
problem is now rereading the directory which we know changed but 
probably only changed a little bit.  So the idea is to create a
READDIR_CHANGES op which takes the opaque directory state cookie.  If
the server does not recognize it or does choose to honor it, it gives
you a flag that says so and then just what a READDIR would.  The server
can freely get rid of state associated with such cookies whenever it
feels like it (and reboot a is time it might definitely feel like 
it :-).

However, if the server does recognize the cookie it just gives the
changes from the state denoted by that cookie to the present.  Only
directory entries which have changed or been deleted (suitably marked)
or for which any of the requested attributes have changed must be
sent but the server can send any he isn't sure of.  It isn't required
that the entry must actually have changed.

I think this would fairly simply address the VLRMD problem and servers
can have a range of choices about how much effort to put into saving
such state depending on experience and the kinds of workloads that 
need performance help.

-----Original Message-----
From: Khan, Saadia 
Sent: Monday, April 18, 2005 8:14 PM
To: [email-masked]
Subject: [nfsv4] Directory delegations and notifications

Now that we are moving forward with the NFSV4.1 draft, I think its about
time we reach a consensus on what should be added to NFSV4.1 regarding
directory delegations. I believe there is general agreement in the group
that read directory delegations are useful and a number of people have
shown interest in implementing them if they were part of NFSV4.1. 

The main issues of contention are change notifications and write
directory delegations and I would like to get some feedback from the
group on how to proceed with those, hence this email.

For large directories which are read-only, read directory delegations
are the solution. However in a real world where directories do change,
without some way of notifying clients about the changes, delegations
will not be of much use since they will get recalled every so often
causing the clients to re-read the whole directory. This is especially
true for very large read-mostly directories. Notifications however do
not make sense in workloads where the directories are changing very
frequently and can flood the network very quickly. In such cases, it
would be recommended not to request notifications. 

As for write directory delegations, they have been proposed to increase
the ability of clients to issue directory operations asynchronously and
decrease the amount of file delegation state that the server needs to
hold. However there are still outstanding design issues that need to be
resolved. This is in addition to the added implementation complexity
that comes with keeping state locally on the client and flushing it to
the server when a request comes in that requires that.

It will be good to know if there is strong support for either one, both
or neither of these functionalities, so that we can move forward. Also
if there are thoughts on splitting delegations and notifications into
two separate drafts or making them two different operations, that would
be good to know as well. Any other feedback/suggestions are also
appreciated.

Thanks.
Saadia

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brian Wickman <[email-masked]>
Date: Tue, 19 Apr 2005 12:36:34 -0400 (EDT)

> add to the complications.

I addressed the fileid issue in my personal draft.  You say the file
presumably has to be statable at create, but this is not necessarily the
case.  When you issue an open() in POSIX, you're given a filehandle and
you're very much allowed to start issuing reads and writes without stating
that file.  There's a grace period in which the client can act as a proxy
for the OPEN, namely, the time between OPEN and the first stat.

What is NOT trivially solvable is the filename issue.  Because servers may
implement arbitrarily complex behavior here, I'm not sure if there's a
good v4.1 time-frame solution.  Perhaps a survey on behavior of server
implementations is appropriate at this time to see if the behavior can be
quantified in any reasonable manner.

> so that the client can do what the server would have done.

A good question here is how often REMOVE/RENAME occurs on an open file.
It could easily be mandated within the spec that these ops be called
synchronously.  I think the majority of benefit would still be there.

> feels like it (and reboot a is time it might definitely feel like
....
> such state depending on experience and the kinds of workloads that

I had this same idea as well, but never came up with a clear way of
implementing it.  I'm very much in favor of doing this, because I think
the savings would be great.  I have data to support that, in conjunction
with read-only directory delegations, the amount of traffic necessary to
communicate changes of directory contents after a delegation revocation is
several orders of magnitude less than required by notifications *in the
common case*.

Cheers,
Brian

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 19 Apr 2005 14:18:04 -0400

I work on DEC/Digital/Compaq/HP's Tru64 Unix, so I and it are pretty
irrelevant to the discussion, but I'll pipe up anyway.

Brian Wickman wrote:

A number of Unix clients do VOP_GETATTR() calls outside of stat(2) calls
by applications.  Not knowing the innards of an NFS V4 implementation,
I don't know if that's a challenge for some clients.  It would be for
Tru64 as the code currently stands.

>From Tru64 Unix's tmpfile(3):

  The tmpfile() function creates a temporary file and returns its FILE
  pointer. The file is opened for update. The temporary file is automatically
  deleted when the process using it terminates.

The pertinent code, note the unlink:

FILE  *
tmpfile(void)
{
	char	tfname[L_tmpnam];
	FILE	*p = NULL;
	int fd;

	(void) tmpnam(tfname);
	if ((fd = open(tfname, O_RDWR|O_EXCL|O_CREAT, 0666)) != -1) {
		p = fdopen(fd, "wb+");
		(void) unlink(tfname);
	}
	return(p);
}

This appears to be the way the code came with OSF/1 Release 1.0.

My Linux system's man page says essentially the same thing, so I
suspect their code does the same thing.

Removing the tempfile's dir entry right after open is a handy way of
reducing the number of stray temp files due to crashes or ^Cs.

	-Ric Werme
-- 
Eric (Ric) Werme         |  [email-masked]
Hewlett-Packard Co.      |  http://werme.8m.net/