Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues

From: David Noveck <[email-masked]>
Date: Thu, 2 Jul 2020 06:06:55 -0400

On Tuesday, June 30, 2020, Trond Myklebust <[email-masked]> wrote:

> On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
>
> Thanks for your helpful comments.
>
> On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <[email-masked]>
> wrote:
>
> Like it or not, the readdir cookie *is* an attribute of the directory.
>
>
> If the protocol treated them as such, then the attribute notifications
> feature could provide updates to the client.   Given that it doesn't, we
> could add a cookie update feature to directory notification feaure as a
> v4.2 extension to the protocol.  However, I'm reluctant to start work on
> the necessary protocol additions until we are sure they are needed to
> provide better directory cacheability.
>
> Actually, they are attributes of directory *streams.   *The difference is
> not all that important given that client implementations are unlikely to be
> aware of the specific steam associated with any particular request.
> However, there are a few cases in which the difference is important in
> determining whether various approaches to client handling of cookies might
> or might not work, and will be important in the discussion below:
>
>    - Two requests made on different clients necessarily are made on
>    distinct streams.
>    - Two requests made on different instances of the same client (with an
>    intervening restart/reboot) also have to arise on different streams.
>
>
> If I want to support the POSIX telldir() and seekdir() operations (
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/seekdir.html ),
> then I need to ensure that when the application calls seekdir(), I return
> to the exact same cursor location in the stream that I was at when I called
> telldir().
>
>
> Agreed.
>
> Without a server side cookie on which to anchor my telldir() cookies,
>
>
> Every client has these available but it is not clear to me useful such
> anchoring is.   I think the flexibility that each client has to
> assign cookies to streams it is responsible for is valuable and could be
> compromised if anchoring to the server cookies is made the focus of the
> implementation.
>
> then all I have is a list of filenames that can and will change every time
> a file is created, deleted or renamed.
>
>
> Clearly it will change.  However, the directory notifications feature
> makes some assumptions, currently implicit about how the list will change.
>  Once these are made explicit, the wg could decide that server/fs pairs
> incapable of staying within these reasonable restrictions (if they are, in
> fact, reasonable), cannot support the directory notifications feature.
>
> Both the length and ordering of that list may change whenever the
> directory is modified,
>
>
> Clearly the length will change, but the reasonable expectation is that
> creating a file will increase the length by one and deleting one will
> decrease it by one.   I don't see the value of supporting directory
> notifications on server fs that do something else.
>
> With regard ro ordering,  suppose the spec allows an fs to shuffle the
> directory order every time a change is made, but I'm unaware of any actual
> file systems that do this.   Do we need to support directory notifications
> for such fs's?
>
>
>
> touch foo; touch bar; ln foo baz; rm foo; mv baz foo
>
> There... Most filesystems will end up reordering 'foo' and 'bar' in the
> directory stream given the above sequence of commands. How does the client
> figure out what happened if the above sequence of commands is performed on
> the server?
> Now let's say that is a directory of a million files, and something like
> the above is made to happen regularly. How do I maintain a stable list of
> synthetic cookies on the client?
>
> I think you are right about there being cases in which it is impossible,
but we either disagree or are simply talking past one another about other
cases.

If the caching client is making the directory changes, then I agree this
cannot be done and you are stuck having to refetch potentially large
directories to deal with new READDIR requests☹️

Where we might disagree is the case in which another client is making the
change.  In that case directory notifications would allow you to avoid
repeated READDIR ops, whether you are providing the user synthetic or
server-based cookies.

My talk on directory caching will discuss the possibility of v4.2
extensions to address the same-client directory caching issue, as well as
possible clarifications regarding directory delegation/notification in v4.1.

>
> meaning that a naive implementation
>
>
> OK.   I'll plead guilty to one *misdemeanor *count of directory naivety.
>
>
> of synthetic cookies as an offset is not compatible with the
> telldir()/seekdir() requirements.
>
>
> It's not clear to me how this incompatibility would manifest itself.  I
> think I need to understand what would break.
>
>
> To make matters worse, the list size is for all intents and purposes
> unbounded, because there is no hard limit on the size of a directory. That
> makes it also impossible to create a cached mapping between a synthetic
> cookie and a filename; such a mapping would be unbounded both in size and
> in duration (since we don't know a priori how long the application will
> keep the directory open, or for that matter, which exact set of cookies it
> may have cached).
>
>
> Such a mapping would, in essence, be part of the cached directory.   So,
> if it is too big to keep in client memory,then it is too big to cache and
> you might as well decide not to cache it.
>
> I expect there is an issue that is a worry in the case in which a
> reasonably sized directory  grows over time to be too big to cache while an
> open directory stream retains some directory cookies which might be
> incompatible with the client dropping  caching of directories and switching
> to server-based cookies.😖
> I feel it is reasonable to treat this situation as one might a
> cookie-verifier failure, particularly if this is the only worrisome failure
> mode.   However, this possibility means that I would not ask clients to
> implement such local cookies. To enable that, we would have to make
> explicit the same sort of reasonableness requirement for cookie changes
> that we have already discussed for ordering changes.  RFC7530 already
> alludes to the need to avoid spurious cookie invalidations although not in
> as explicit or strict way as we would need to support
> directory notifications:
>
>    As there is no way for the client to indicate that a cookie value,
>
>    once received, will not be subsequently used, server implementations
>
>    should avoid schemes that allocate memory corresponding to a returned
>
>    cookie.  Such allocation can be avoided if the server bases cookie
>
>    values on a value such as the offset within the directory where the
>
>    scan is to be resumed.
>
>
>    Cookies generated by such techniques should be designed to remain
>
>    valid despite modification of the associated directory.  If a server
>
>    were to invalidate a cookie because of a directory modification,
>
>    READDIRs of large directories might never finish.
>
>
>
> So in order to make this work the client would basically have to create
> its own B-tree and persist it in storage somewhere.
>
>
> I don't see the need to make this persistent.  If the client restarts, all
> directory streams have ceased to exist and we know  *a posteriori  *that
> there are no outstanding directory cookies to which the client would have
> to respond.
>
> <[email-masked]>
>
> --
>
> --
>
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [email-masked] <[email-masked]>
>
>
>

---

From: Rick Macklem <[email-masked]>
Date: Fri, 3 Jul 2020 01:03:09 +0000

Well, I'll throw out a simple suggestion w.r.t. directory cookies.

What if the cookies generated by the server were required to have the
following properties:
- Monotonically increasing numeric values.
- Sparse enough that new entries can be created with numeric values
  between them.
- Guaranteed to not change and remain valid (and referring to the same entry)
  despite additions/deletions. (Until a deletion callback is received by the client.)

For example, for a simple case of a UFS file system on a server, the UFS
directory consists of blocks of directory entries.
- When an entry is added, it goes in a gap within the directory or grows a
   new block at the end (if I vaguely recall this correctly;-).
- When an entry is removed, the entry is erased without moving other
   entries.
For this example, the directory offset cookie can simply be the byte offset
of the entry.
(I haven't looked at other file systems, but hopefully offset cookies with the
 above properties can be created for most of them?)

When an entry is added/deleted, the server issues a callback to the client
with the new directory cookie offset (and the directory entry for the add case).

The client can maintain this structure in any number of ways (and it could be
fun figuring out what works well), but a trivial version could be:
- The reply to a readdir is kept as a list head (with the directory offset of the
   first entry) and a linked list of the entries in order, with their cookie.
  (Remember that the cookies are in the same ordering as the entries.)
- The next readdir reply creates the next head/list.

readdir() just works through each list, following each head in order.
telldir() returns the cookie for the entry.
seekdir() just finds the correct list and then searches down that list for a match.

The remove/add entry callbacks just insert/delete entrie(s) in the appropriate
list.
(I'd probably keep these lists in the kernel client under the VFS for FreeBSD,
 as malloc'd data structures, but that is simply an implementation choice.)

You would require the monotonically increasing property for directory
delegations to be issued. (Without that you don't know where to insert
additions.) You would also require that extant directory entry cookies
remain valid and unchanged when additions/deletions occur.
(Note that removing an entry and then adding an entry at the same offset
 is allowed under POSIX telldir()/seekdir() as I understand it.)

This avoids any need for the client to synthesize cookies and just use the ones
returned by the server, I think?

Just a simple idea that may be worth considering?
(If this has already been discussed, I apologize for not seeing it.)

rick

________________________________________
From: nfsv4 <[email-masked]> on behalf of David Noveck <[email-masked]>
Sent: Thursday, July 2, 2020 6:06 AM
To: Trond Myklebust
Cc: [email-masked]
Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to [email-masked]

On Tuesday, June 30, 2020, Trond Myklebust <[email-masked]<mailto:[email-masked]>> wrote:
On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
Thanks for your helpful comments.

On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <[email-masked]<mailto:[email-masked]>> wrote:
Like it or not, the readdir cookie is an attribute of the directory.

If the protocol treated them as such, then the attribute notifications feature could provide updates to the client.   Given that it doesn't, we could add a cookie update feature to directory notification feaure as a v4.2 extension to the protocol.  However, I'm reluctant to start work on the necessary protocol additions until we are sure they are needed to provide better directory cacheability.

Actually, they are attributes of directory streams.   The difference is not all that important given that client implementations are unlikely to be aware of the specific steam associated with any particular request.  However, there are a few cases in which the difference is important in determining whether various approaches to client handling of cookies might or might not work, and will be important in the discussion below:

  *   Two requests made on different clients necessarily are made on distinct streams.
  *   Two requests made on different instances of the same client (with an intervening restart/reboot) also have to arise on different streams.

If I want to support the POSIX telldir() and seekdir() operations ( https://pubs.opengroup.org/onlinepubs/9699919799/functions/seekdir.html ), then I need to ensure that when the application calls seekdir(), I return to the exact same cursor location in the stream that I was at when I called telldir().

Agreed.

Every client has these available but it is not clear to me useful such anchoring is.   I think the flexibility that each client has to assign cookies to streams it is responsible for is valuable and could be compromised if anchoring to the server cookies is made the focus of the implementation.

then all I have is a list of filenames that can and will change every time a file is created, deleted or renamed.

Clearly it will change.  However, the directory notifications feature makes some assumptions, currently implicit about how the list will change.   Once these are made explicit, the wg could decide that server/fs pairs incapable of staying within these reasonable restrictions (if they are, in fact, reasonable), cannot support the directory notifications feature.

Both the length and ordering of that list may change whenever the directory is modified,

Clearly the length will change, but the reasonable expectation is that creating a file will increase the length by one and deleting one will decrease it by one.   I don't see the value of supporting directory notifications on server fs that do something else.

With regard ro ordering,  suppose the spec allows an fs to shuffle the directory order every time a change is made, but I'm unaware of any actual file systems that do this.   Do we need to support directory notifications for such fs's?

There... Most filesystems will end up reordering 'foo' and 'bar' in the directory stream given the above sequence of commands. How does the client figure out what happened if the above sequence of commands is performed on the server?
Now let's say that is a directory of a million files, and something like the above is made to happen regularly. How do I maintain a stable list of synthetic cookies on the client?

I think you are right about there being cases in which it is impossible, but we either disagree or are simply talking past one another about other cases.

If the caching client is making the directory changes, then I agree this cannot be done and you are stuck having to refetch potentially large directories to deal with new READDIR requests☹️

Where we might disagree is the case in which another client is making the change.  In that case directory notifications would allow you to avoid repeated READDIR ops, whether you are providing the user synthetic or server-based cookies.

My talk on directory caching will discuss the possibility of v4.2 extensions to address the same-client directory caching issue, as well as possible clarifications regarding directory delegation/notification in v4.1.

OK.   I'll plead guilty to one misdemeanor count of directory naivety.

of synthetic cookies as an offset is not compatible with the telldir()/seekdir() requirements.

It's not clear to me how this incompatibility would manifest itself.  I think I need to understand what would break.

To make matters worse, the list size is for all intents and purposes unbounded, because there is no hard limit on the size of a directory. That makes it also impossible to create a cached mapping between a synthetic cookie and a filename; such a mapping would be unbounded both in size and in duration (since we don't know a priori how long the application will keep the directory open, or for that matter, which exact set of cookies it may have cached).

Such a mapping would, in essence, be part of the cached directory.   So, if it is too big to keep in client memory,then it is too big to cache and you might as well decide not to cache it.

I expect there is an issue that is a worry in the case in which a reasonably sized directory  grows over time to be too big to cache while an open directory stream retains some directory cookies which might be incompatible with the client dropping  caching of directories and switching to server-based cookies.😖
I feel it is reasonable to treat this situation as one might a cookie-verifier failure, particularly if this is the only worrisome failure mode.   However, this possibility means that I would not ask clients to implement such local cookies. To enable that, we would have to make explicit the same sort of reasonableness requirement for cookie changes that we have already discussed for ordering changes.  RFC7530 already alludes to the need to avoid spurious cookie invalidations although not in as explicit or strict way as we would need to support directory notifications:

   scan is to be resumed.

So in order to make this work the client would basically have to create its own B-tree and persist it in storage somewhere.

I don't see the need to make this persistent.  If the client restarts, all directory streams have ceased to exist and we know  a posteriori  that there are no outstanding directory cookies to which the client would have to respond.
<mailto:[email-masked]>

--

--

Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email-masked]<mailto:[email-masked]>

---

From: David Noveck <[email-masked]>
Date: Fri, 3 Jul 2020 06:43:42 -0400

> Well, I'll throw out a simple suggestion w.r.t. directory cookies.

I'm also working on a suggestion about helping clients maintain directory
cookies. Slides will be presented 7/9.

> What if the cookies generated by the server were required to have the

I don't think we can add new requirements for directory cookies.
Requirements for the granting or retention of directory delegations are
possible but I'd like  to derive them from inherent aspects of the existing
notification scheme. Adding wholly new requirements on an existing feature
is not possible.

Not possible to add this requirment. Also don't see the need/value of
this.  In my view, the existing directory notification scheme provides
adequate information for subscribed clients to maintain the structure of
cached directories being modified, including cookie values.

>   between them.

Again, a new requirement, with no clear justification.

- Guaranteed to not change and remain valid (and referring to the same
> entry)

Should also be allowed if the directory delegation is recalled.

I'm unclear about the callbacks you are proposing. Normally, notification
callbacks are not sent to the client making the change. It might be
possible to change that in a v4.2 extension but it adds to operation
latency and creates sequencing difficulties ☹️

I considered returning the necessary info in a bunch of new ops but the
whole thing got waaay too complicated.  I think the best choice would be a
GET_NOTIFICATION op to be issued after the CREATE, OPEN, LINK, REMOVE,
RENAME.

>  above properties can be created for most of them?)

Possibly, but given the need for this, I'm not inclined to rely on hope.

> with the new directory cookie offset (and the directory entry for the add

I'd prefer that the same information that is returned as in the existing
directory notification scheme, which allows you to maintain directory order
and cookies, without tieing one to the other.

>   (Remember that the cookies are in the same ordering as the entries
>

Not in my scheme.

> additions.)

The insert notification gives you enough information to do this without the
monotonically increasing property.

You would also require that extant directory entry cookies
>  is allowed under POSIX telldir()/seekdir() as I understand it.)

I think the requirement goes away when the delegation does.

> returned by the server, I think?

Yes

> (If this has already been discussed,

Don't think it has.

I apologize for not seeing it.)

Even if it had, no apology would be necessary

> CAUTION: This email originated from outside of the University of Guelph.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe. If in doubt, forward suspicious emails to
> [email-masked]
>
>
>
> Actually, they are attributes of directory streams.   The difference is
>   *   Two requests made on different instances of the same client (with an
> https://pubs.opengroup.org/onlinepubs/9699919799/functions/seekdir.html
> ), then I need to ensure that when the application calls seekdir(), I
> return to the exact same cursor location in the stream that I was at when I
> called telldir().
>
>
> Agreed.
>
> anchoring is.   I think the flexibility that each client has to assign
> cookies to streams it is responsible for is valuable and could be
> as explicit or strict way as we would need to support directory
> notifications:
>
> directory streams have ceased to exist and we know  a posteriori  that

---

From: Rick Macklem <[email-masked]>
Date: Fri, 3 Jul 2020 17:09:24 +0000

David Noveck wrote:
>Rick Macklem  wrote:
>I'm also working on a suggestion about helping clients maintain directory cookies. >Slides will be presented 7/9.
>
>
>I don't think we can add new requirements for directory cookies.  Requirements for >the granting or retention of directory delegations are possible but I'd like  to >derive them from inherent aspects of the existing notification scheme. Adding >wholly new requirements on an existing feature is not possible.
>
- Without the monotonically increasing values for cookies, the client side
  implementation becomes more difficult. To insert entries efficiently, it would
  have to find the extant entry that a new entry follows. To do this is going to
  require some sort of data structure that maintains the ordering of the entries
  and allows for efficient search based on a "random" cookie.
  --> Too difficult for client implementors --> Never gets implemented.

>Not possible to add this requirment. Also don't see the need/value of this.  In my >view, the existing directory notification scheme provides adequate information for >subscribed clients to maintain the structure of cached directories being modified, >including cookie values.
Ok, so if it cannot be a requirement, what about an optional property?
- Add a new read-only, per-file system attribute that is a Boolean that states
  whether or not the directory cookies generated by the server have the property.
  --> Then the client can choose whether or not to try and use directory
         delegations based on that.

>- Sparse enough that new entries can be created with numeric values
As worded, this is just a suggestion for server implementors.
The optional property would be invariance of extant cookies and monotonically
increased ordering maintained.

>Again, a new requirement, with no clear justification.
Justification, as above.

>Should also be allowed if the directory delegation is recalled.
Yes. 

>I'm unclear about the callbacks you are proposing. Normally, notification callbacks >are not sent to the client making the change. It might be possible to change that >in a v4.2 extension but it adds to operation latency and creates sequencing >difficulties ☹️
I'll admit I had not read it in a long time and my recollection was similar.
I just re-read it and it states that the client doing the addition/deletion gets
the CB_NOTIFY. Here's the snippet:
   If the client holding the delegation makes any changes in the
   directory that cause files or sub-directories to be added or removed,
   the server will notify that client of the resulting change(s).  If
   the client holding the delegation is making attribute or cookie
   verifier changes only, the server does not need to send notifications
   to that client.  The server will send the following information for
   each operation:

I was mistaken and thought the CB_NOTIFY did not go to the client that did
the addition/deletion.
Since the CB_NOTIFY does go to the client doing the addition/deletion, I
think the 4.1 CB_NOTIFY is sufficient.

>I considered returning the necessary info in a bunch of new ops but the whole >thing got waaay too complicated.  I think the best choice would be a >GET_NOTIFICATION op to be issued after the CREATE, OPEN, LINK, REMOVE, >RENAME.
>
As above, the client doing the addition/deletion does get the CB_NOTIFY.

>Possibly, but given the need for this, I'm not inclined to rely on hope.
If the server cannot do it, the new read-only attribute would be false and
(at least the FreeBSD client) would choose not to get directory delegations.

>I'd prefer that the same information that is returned as in the existing directory >notification scheme, which allows you to maintain directory order and cookies, >without tieing one to the other.
I was just being lazy. I hadn't read the section of RFC5661 in a long time and
didn't remember the terminology. (I didn't mean to imply "new callbacks" were
needed.) There was the issue w.r.t the client doing the addition/deletion
getting a callback (which it appears it does).

>>  (Remember that the cookies are in the same ordering as the entries
Well, I do this for fun. If it isn't fun, I won't be doing it. (I know someone
doing this as a retirement hobby is weird, but since vendors do not
invest in clients much...)

>The insert notification gives you enough information to do this without the >monotonically increasing property.
That is true, but then why hasn't anyone implemented it?
(Yes, I was "misusing" the term require. I'm not a spec. writer, as you can easily tell.)

>I think the requirement goes away when the delegation does.
Yes, although it practice, assuming the client(s) are getting the notifications,
I only see them returning a directory delegation after a closedir() and if the
number of CB_NOTIFYs is large or the caching storage needs to be free'd.

>Even if it had, no apology would be necessary
I should have re-read the appropriate sections of RFC5661 before the last post.

rick

rick

________________________________________
From: nfsv4 <[email-masked]<mailto:[email-masked]>> on behalf of David Noveck <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, July 2, 2020 6:06 AM
To: Trond Myklebust
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to [email-masked]<mailto:[email-masked]>

On Tuesday, June 30, 2020, Trond Myklebust <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>> wrote:
On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
Thanks for your helpful comments.

On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>> wrote:
Like it or not, the readdir cookie is an attribute of the directory.

Agreed.

   scan is to be resumed.

I don't see the need to make this persistent.  If the client restarts, all directory streams have ceased to exist and we know  a posteriori  that there are no outstanding directory cookies to which the client would have to respond.
<mailto:[email-masked]<mailto:[email-masked]>>

--

--

Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>

---

From: David Noveck <[email-masked]>
Date: Sat, 4 Jul 2020 07:09:03 -0400

On Fri, Jul 3, 2020, 1:09 PM Rick Macklem <[email-masked]> wrote:

> cookies. >Slides will be presented 7/9.
> >
> >
> Requirements for >the granting or retention of directory delegations are
> possible but I'd like  to >derive them from inherent aspects of the
> existing notification scheme. Adding >wholly new requirements on an
> existing feature is not possible.
> >
> this.  In my >view, the existing directory notification scheme provides
> adequate information for >subscribed clients to maintain the structure of
> cached directories being modified, >including cookie values.

That's not a problem for me and feel it should be added given it would
enable client implementation.  It seems likely that most server
implementations would be able to provide cookies with this property.

> callbacks >are not sent to the client making the change. It might be
> possible to change that >in a v4.2 extension but it adds to operation
> latency and creates sequencing >difficulties ☹️

Now it's my turn to apologise.  I obviously was not reading carefully
enough when I discussed this.  I thought I had seen text saying the
contrary, but when I went back and looked for it, it wasn't there.

> think the 4.1 CB_NOTIFY is sufficient.

My feeling is that it is barely sufficient but we will need to discuss the
possible need for further extensions at the meeting and on the list. The
reasons I'm not sanguine about this approach are:

   - It is a problem for latency,which is a concern for common operations
   like OPEN-create.
   - There are troubling synchronization problems, given that the
   notification can be received before the request response or seconds after.
   - Unless you are prepared to lock the directory on the client until the
   callback is received (not so easy to do since the callback is asynchronous,
   you face an even worse synchronization problems as you receive multiple
   notifications (in-order but not organized by request).

If we want to do this in 4.1 (e.g. Linux is OK without cookie
monotonicity), then we are going have to fix up issues with regard to
dirent_notiy_delay.  This is the  minimum number of *seconds *to delay
sending the notification.   We'd have to change this so that it didn't
apply to the same client case.  It pretty clearly is designed for the
other-client case.  I'm pretty sure we could do this in the rfc5661bis
context, along with some other cleanup in this area.

> whole >thing got waaay too complicated.  I think the best choice would be a
> directory >notification scheme, which allows you to maintain directory
> order and cookies, >without tieing one to the other.

> From: nfsv4 <[email-masked]<mailto:[email-masked]>> on
> behalf of David Noveck <[email-masked]<mailto:[email-masked]>>
> [email-masked]<mailto:[email-masked]>
>
>
>
> On Tuesday, June 30, 2020, Trond Myklebust <[email-masked]
> <mailto:[email-masked]><mailto:[email-masked]<mailto:
> [email-masked]>>> wrote:
> On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <[email-masked]
> <mailto:[email-masked]><mailto:[email-masked]<mailto:
> [email-masked]>>> wrote:
> [email-masked]<mailto:[email-masked]
> ><mailto:[email-masked]<mailto:
> [email-masked]>>
>
>
>
>

---

From: Rick Macklem <[email-masked]>
Date: Sun, 5 Jul 2020 01:33:05 +0000

David Noveck wrote:
[stuff snipped]
>My feeling is that it is barely sufficient but we will need to discuss the possible
> need for further extensions at the meeting and on the list. The reasons
> I'm not sanguine about this approach are:
>
>  *   It is a problem for latency,which is a concern for common operations
> like OPEN-create.
>  *   There are troubling synchronization problems, given that the

Yes. Have you read the first para of RFC5661 sec. 18.39.4 lately:
   Directory delegations provide the benefit of improving cache
   consistency of namespace information.  This is done through
   synchronous callbacks.  A server must support synchronous callbacks
   in order to support directory delegations.  In addition to that,
   asynchronous notifications provide a way to reduce network traffic as
   well as improve client performance in certain conditions.

When I read this, I have no idea what "synchronous callbacks" refers to?
It almost hints that certain callbacks are done synchronously and then
others asynchronously. Maybe the original author was trying to address
this problem. Could it be that "asynchronous notifications" was not meant
to be all the notifications?

>  *   Unless you are prepared to lock the directory on the client until the callback
> is received (not so easy to do since the callback is asynchronous, you face
> an even worse synchronization problems as you receive multiple
>If we want to do this in 4.1 (e.g. Linux is OK without cookie monotonicity),
> then we are going have to fix up issues with regard to dirent_notiy_delay.

I would be interested in hearing if having monotonically increasing directory
offset cookies would make the Linux client implementation easier?

>  This is the minimum number of seconds to delay sending the notification.
>   We'd have to >change this so that it didn't apply to the same client case.
>  It pretty clearly is designed for the other-client case.
>  I'm pretty sure we could do this in the rfc5661bis context, along
> with some other cleanup in this area.

Actually, my reading of RFC5661 is that dirent_notif_delay only applies to
attribute notifications. See this para in pg 582:
   NOTIFY4_CHANGE_CHILD_ATTRS/NOTIFY4_CHANGE_DIR_ATTRS
      The client will use the attribute mask to inform the server of
      attributes for which it wants to receive notifications.  This
      change notification can be requested for changes to the attributes
      of the directory as well as changes to any file's attributes in
      the directory by using two separate attribute masks.  The client
      cannot ask for change attribute notification for a specific file.
      One attribute mask covers all the files in the directory.  Upon
      any attribute change, the server will send back the values of
      changed attributes.  Notifications might not make sense for some
      file system-wide attributes, and it is up to the server to decide
      which subset it wants to support.  The client can negotiate the
      frequency of attribute notifications by letting the server know
      how often it wants to be notified of an attribute change.  The
      server will return supported notification frequencies or an
      indication that no notification is permitted for directory or
      child attributes by setting the dir_notif_delay and
      dir_entry_notif_delay attributes, respectively.
Called dir_entry_notif_delay here, just to try and confuse us;-)
Since attribute change notifies do not go to the client that created/removed
the entry, I think the above is addressed?

That brings us back to "synchronous callbacks". Although it is not clear,
I might contend that the original author might have intended that the
NOTIFY4_ADD_ENTRY/NOTIFY4_REMOVE_ENTRY/NOTIFY4_RENAME_ENTRY
be done synchronously?

Anyhow, I agree that this needs clarification.

rick

rick

rick

________________________________________
From: nfsv4 <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>> on behalf of David Noveck <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>
Sent: Thursday, July 2, 2020 6:06 AM
To: Trond Myklebust
Cc: [email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>
Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to [email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>

On Tuesday, June 30, 2020, Trond Myklebust <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>> wrote:
On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
Thanks for your helpful comments.

On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>> wrote:
Like it or not, the readdir cookie is an attribute of the directory.

Agreed.

   scan is to be resumed.

I don't see the need to make this persistent.  If the client restarts, all directory streams have ceased to exist and we know  a posteriori  that there are no outstanding directory cookies to which the client would have to respond.
<mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

--

--

Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

---

From: David Noveck <[email-masked]>
Date: Sun, 5 Jul 2020 07:22:01 -0400

> Yes. Have you read the first para of RFC5661 sec. 18.39.4 lately:

No.

>    well as improve client performance in certain conditions.

Thanks for digging into this.  I appreciate you efforts to shed light on
this, especially since I find it unlikely that they were "fun".

> When I read this, I have no idea what "synchronous callbacks" refers to?

My interpretation is that the server waits for the callback response before
sending the reply. It makes sense but I'm not sure it's right.

> others asynchronously.

It does more than hint.  It implies it strongly.

Maybe the original author was trying to address
> this problem.

Yes.

Could it be that "asynchronous notifications" was not meant
> to be all the notifications?

It could.

> monotonicity),
> attribute notifications.

I don't agree. See below.

>       dir_entry_notif_delay attributes, respectively.

I think you are reading too much into the fact that this the only place
that these delays are explicitly mentioned.  Maybe I'm relying too much on
the fact that delay/batching of directory entry changes makes sense in the
other-client case.

> NOTIFY4_ADD_ENTRY/NOTIFY4_REMOVE_ENTRY/NOTIFY4_RENAME_ENTRY

Might have but given the possibility that a number of clients might hold
delegations, issues regarding unresponsive clients would have to be
addressed to go down this path.

Another possibility is that it was intended that callbacks to the same
client were intended to be synchronous.

I'll do my best to involve the original author In the discussion.

> From: nfsv4 <[email-masked]<mailto:[email-masked]><mailto:
> [email-masked]<mailto:[email-masked]>>> on behalf of
> David Noveck <[email-masked]<mailto:[email-masked]><mailto:
> [email-masked]<mailto:[email-masked]>>>
> [email-masked]<mailto:[email-masked]><mailto:[email-masked]
> <mailto:[email-masked]>>
>
>
>
> [email-masked]>><mailto:[email-masked]<mailto:
> [email-masked]><mailto:[email-masked]<mailto:
> [email-masked]>>>> wrote:
> [email-masked]>><mailto:[email-masked]<mailto:
> [email-masked]><mailto:[email-masked]<mailto:
> [email-masked]>>>> wrote:
> ><mailto:[email-masked]<mailto:
> [email-masked]>><mailto:[email-masked]
> <mailto:[email-masked]><mailto:
> [email-masked]<mailto:[email-masked]>>>
>
>
>
>

---

From: Rick Macklem <[email-masked]>
Date: Sun, 5 Jul 2020 22:59:07 +0000

[stuff snipped]
>Thanks for digging into this.  I appreciate you efforts to shed light on this,
> especially since I find it unlikely that they were "fun".
Kinda interesting, which can make it fun. Recently on a FreeBSD mailing list
there was a discussion about poor performance accessing a directory with
10million entries over NFS, so at least some care about this.

>>It almost hints that certain callbacks are done synchronously and
>> then others asynchronously.
>
(To correct this, I meant dir_notif_delay, not direny_notif_delay.)
>I think you are reading too much into the fact that this the only place that these >delays are explicitly mentioned.  Maybe I'm relying too much on the fact that >delay/batching of directory entry changes makes sense in the other-client case.
Well, there is also the definition of dir_notif_delay:
5.11.1.  Attribute 56: dir_notif_delay

   The dir_notif_delay attribute is the minimum number of seconds the
   server will delay before notifying the client of a change to the
   directory's attributes.

Although Trond did say that the directory offset was an attribute, I would have
assumed "attribute" in the above sentence referred to fattr4 items.

Now, I am not sure about batched notifies, but it could be that the author
was thinking of multiple changes caused by the same compound RPC?

> addressed to go down this path.
That is always going to be a problem. When a server receives a conflicting Open,
file delegations need to be CB_RECALL'd. All the server can do is try for a while
and then give up.

>Another possibility is that it was intended that callbacks to the same client
> were intended to be synchronous.
I think this might be a desirable trait. I think the client that has done the RPC
that adds/removes a name from the directory will need to "wait" until the
add/remove notification arrives before unlocking the directory for reading.
For example, for "add", if it does not keep the directory locked,
the entry will not be seen by a readdir(), although it exists and can be Open'd.
As such, I think the delay should be minimal for this client.
I think it is somewhat less critical for the other client(s) that hold
a delegation on the directory.

This para, from sec. 10.9.2 seems relevant:
   In addition to asking for delegations, a client can also ask for
   notifications for certain events.  These events include changes to
   the directory's attributes and/or its contents.  If a client asks for
   notification for a certain event, the server will notify the client
   when that event occurs.  This will not result in the delegation being
   recalled for that client.  The notifications are asynchronous and
   provide a way of avoiding recalls in situations where a directory is
   changing enough that the pure recall model may not be effective while
   trying to allow the client to get substantial benefit.  In the
   absence of notifications, once the delegation is recalled the client
   has to refresh its directory cache; this might not be very efficient
   for very large directories.

Although it states the notifications are asynchronous, it also states the
notifications are done "when that even occurs". It does not indicate that
there may be a significant delay before doing the notification.

rick

rick

rick

________________________________________
From: nfsv4 <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>> on behalf of David Noveck <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>>
Sent: Thursday, July 2, 2020 6:06 AM
To: Trond Myklebust
Cc: [email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>
Subject: Re: [nfsv4] Notes regarding discussion of directory scalabiliy issues

CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to [email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>

On Tuesday, June 30, 2020, Trond Myklebust <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>>> wrote:
On Tue, 2020-06-30 at 08:40 -0400, David Noveck wrote:
Thanks for your helpful comments.

On Mon, Jun 29, 2020 at 2:43 PM Trond Myklebust <[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>>> wrote:
Like it or not, the readdir cookie is an attribute of the directory.

Agreed.

   scan is to be resumed.

I don't see the need to make this persistent.  If the client restarts, all directory streams have ceased to exist and we know  a posteriori  that there are no outstanding directory cookies to which the client would have to respond.
<mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>>

--

--

Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>><mailto:[email-masked]<mailto:[email-masked]><mailto:[email-masked]<mailto:[email-masked]>>>>