Subject: [nfsv4] Directory delegations and notifications

From: "Khan, Saadia" <[email-masked]>
Date: Mon, 18 Apr 2005 17:14:03 -0700

Now that we are moving forward with the NFSV4.1 draft, I think its about
time we reach a consensus on what should be added to NFSV4.1 regarding
directory delegations. I believe there is general agreement in the group
that read directory delegations are useful and a number of people have
shown interest in implementing them if they were part of NFSV4.1. 

The main issues of contention are change notifications and write
directory delegations and I would like to get some feedback from the
group on how to proceed with those, hence this email.

For large directories which are read-only, read directory delegations
are the solution. However in a real world where directories do change,
without some way of notifying clients about the changes, delegations
will not be of much use since they will get recalled every so often
causing the clients to re-read the whole directory. This is especially
true for very large read-mostly directories. Notifications however do
not make sense in workloads where the directories are changing very
frequently and can flood the network very quickly. In such cases, it
would be recommended not to request notifications. 

As for write directory delegations, they have been proposed to increase
the ability of clients to issue directory operations asynchronously and
decrease the amount of file delegation state that the server needs to
hold. However there are still outstanding design issues that need to be
resolved. This is in addition to the added implementation complexity
that comes with keeping state locally on the client and flushing it to
the server when a request comes in that requires that.

It will be good to know if there is strong support for either one, both
or neither of these functionalities, so that we can move forward. Also
if there are thoughts on splitting delegations and notifications into
two separate drafts or making them two different operations, that would
be good to know as well. Any other feedback/suggestions are also
appreciated.

Thanks.
Saadia

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Craig Everhart <[email-masked]>
Date: Tue, 19 Apr 2005 11:05:19 -0400

Along with Dave Noveck, I feel there are too many architectural
issues for write directory delegations to have that working well
for 4.1.

I think that read directory delegations are necessary, though,
and I strongly support them for 4.1.  And I support them whether
there's a special protocol for supporting read-mostly directories
or not (and streaming logged changes back to clients).

> be good to know as well. Any other feedback/suggestions are also

Craig Everhart
+1 919 543 2169

---

From: Carl Burnett <[email-masked]>
Date: Tue, 19 Apr 2005 15:54:45 -0500

I also agree that the major benefits come from read delegations with 
synchronous callbacks. As I recall, the study done by the CITI team with 
some sample University workloads confirmed that. While the notification 
features in Saadia's RFC draft may be interesting in some cases, I do not 
believe they show the same level of compelling value. However, this 
feature is optional in the draft. This can allow initial implementations 
to pursue the basic read with recall model. Follow on implementations can 
experiment with the notification features. From a complexity perspective, 
it don't see a great need to pursue write delegation on directories. As 
another consideration, when applications get a success return code on 
these types of metadata operations (create, delete, etc.), they quite 
often want the guarantee that those events happened successfully on the 
target storage.

I would like to see activity continue forward on Saddia's RFC to bring it 
to a point for consensus, so that vendors can start forming implementation 
plans. 

Thanks,
Carl

Carl Burnett
AIX Kernel Architecture - Network File System
(512) 838-8498, TL 678-8498
(please reply to [email-masked])

Craig Everhart/Pittsburgh/IBM@IBMUS 
Sent by: [email-masked]
04/19/2005 10:05 AM

To
[email-masked]
cc

Subject
Re: [nfsv4] Directory delegations and notifications

> be good to know as well. Any other feedback/suggestions are also

Craig Everhart
+1 919 543 2169_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Brian Wickman <[email-masked]>
Date: Thu, 21 Apr 2005 14:40:23 -0400 (EDT)

I would like to make a few remarks about implications and caveats with
both Saadia's draft and my personal draft.  I agree with Carl and the rest
that directory write delegations are probably not feasible on a v4.1
time-scale because of issues that Dave brought up.  I should point out
that directory write delegations are an optional feature with my draft (in
fact, all delegations are optional features in my draft, because the
server can simply either return ERR_NOTSUPP or just deny all write
delegation requests.)

  There are a few snippets from Saadia's draft about which I'd like to
remark:

 > These extensions allow improved namespace cache consistency to be
 > achieved through delegations and synchronous recalls alone without
 > asking for notifications. In addition, if time-based consistency is
 > sufficient, asynchronous notifications can provide performance
 > benefits for the client, and possibly the server, under some common
 > operating conditions such as slowly-changing and/or very large
 > directories.

  As Carl mentioned, and as some saw at IETF, in the two case studies that
I analyzed, notification traffic (in both bytes and moreso in ops) far
outweighed the traffic necessary to maintain the directory cache -- even
in slowly changing and large directories.  However, this isn't to say that
the problem may exist under certain understood operating conditions at
their extremes.  Under these circumstances, Dave's idea of READDIR_CHANGES
makes a lot more sense.  According to my numbers, on the average, < 5% of
directory entries change between the read directory delegation revocation
and the time the client touches its directory cache again.  In general,
this corresponds to only one directory entry.  I believe this information
is in my IETF slides.

 > ...asynchronous notifications provide a way to reduce network traffic
 > as well as improve client performance in certain conditions.
 > Notifications would not be requested when the goal is just cache
 > consitency.

  First point: The last sentence here makes me pose the question of when
notifications /would/ be requested by the client.  That is, what are the
client best practices regarding requesting notifications?  Also, under
what circumstances will the client also be wanting to hold a directory
read delegation on the same directory?

  Second point: When you want notifications yet don't want notifications
on all possible events, then you get effectively a delegation on the
remaining events (i.e., synchronous recall).  For example, if I request to
get notifications when entries are added, deleted or renamed within the
directory, my notification will be revoked when the directory's attributes
change, say, via a SETATTR.  It makes no sense that notifications should
be recalled by a delegation-like event.  If I'm going to ask for
notifications, I want the ability to choose which notifications I receive
to be meaningful without having to worry about my notifications going away
(being "revoked") from an event that I do not care about.

   Instead, notifications should be recalled when the server feels the
traffic being generated by the notification is far too much for merit.
They shouldn't be tied to the idea of delegation, because it makes it
difficult if not impossible to acquire just notifications.  The most
useful case is to specify no notifications, in which case you get a true
delegation.  In this case, it'd be equivalent to using the operation in my
draft.

  Lastly, delegations are a construct that improve cache consistency.  I
demonstrated this to be true in practice.  Notifications, in the common
case, cause far too much traffic and don't improve upon v2/v3 time-based
consistency.  However, they clearly have use cases but in the uncommon
case, when a given application genuinely would like to see such events and
be notified in user-land about such events (e.g., GUI browser or
producer/consumer system of sorts.)  As such, notifications should be
treated differently from delegations and whether or not they are granted
should be governed by a policy different than that of delegation granting
policy.

  There are no drawbacks of simply having separate GET_DIR_DELEGATION and
GET_DIR_NOTIFICATIONS operation.  It should make the camp that wants
directory delegations happy.  It should make the camp that wants directory
notifications happy.  Having the two constructs tied together reduces the
utility of both, and virtually breaks the utility of notifications
altogether.  Implementing one or the other is optional and you don't need
to implement half of anything to get the utility that you would like.

My two cents.

Cheers,
Brian