Subject: Re: [Gen-art] Gen-art LC review of draft-ietf-nfsv4-rfc3530bis-33

From: Elwyn Davies <[email-masked]>
Date: Mon, 01 Dec 2014 14:41:47 +0000

Hi, Tom and David.

Sorry to have been dilatory in running through the v35 that Tom sent me.

This looks pretty good to me and has sorted out almost everything that I 
flagged up in the review. In particular s7.8 is much improved.

The one problem that I have left is the issue of directory contents 
being changed under the feet of a multi-part readdir request.  The last 
mail on this subject from 14 November said:
>
>
>> On Nov 14, 2014, at 3:05 PM, Tom Haynes <[email-masked]> wrote:
>>

<<snip>>

>>>>> On Nov 11, 2014, at 12:48 PM, Elwyn Davies <[email-masked]
>>>>> <mailto:[email-masked]>> wrote:
>>>>>
>>
>>>>
>>>>>
>>>>> s15.26.4/s15.26.5: What happens if the directory has been modified
>>>>> (entries added or deleted) between calls of READDIR intended to
>>>>> retrieve segments of the directory information?  Related to this,
>>>>> presumably no ordering of entries is implied in the returned
>>>>> information - it would be desirable to state this.
>>>>
>>>>
>>>> This is where the cookieverf is supposed to come in.
>>>>
>>>> In most implementations, the cookie needs to be persistent, which indeed
>>>> means an ordering of the information.
>>>
>>> AFAICS the text doesn't tie the cookieverf to the change attribute of the
 >>> directory.  Presumably this is what is intended - it would 
certainly mean

that changes would be picked up and READDIR would have to start from the 
beginning again.
>>>>
>>
>> Asking an expert…
>
> From Trond:
>
> As far as I know, the cookieverf is about validating the cookies
> themselves. There is no requirement in any of the descriptions in
> RFC1813, RFC3530 or RFC5661 that states that the contents of the
> directory need to change when the cookieverf does or vice-versa.
> In theory, the server could throw out the cookies and regenerate them
> on reboot, for instance.
>

My view (remains) that if the cookie is not tied to the change attribute 
of the directory, this makes life difficult for both the server (what 
does it send if the data has been changed and how does it manage 
maintaining checkpoints -- where a readdir finished --, in particular if 
the checkpoint was on an entry that has been deleted), and the client.

Thoughts?

I also found a some minor typos (mostly finger trouble in the new bits) 
and there are a couple of nits outstanding (see below).

Are you intending to publish a -35 before Thursday?

Regards,
Elwyn

Nits:
=====
s2.1: nfs_lease4 definition still missing.

s8.5, last para: I am not sure that the jargon 'trunking relation' is 
widely understood. [I don't think I have ever come across it.]

s19.4.1, para 2:
 >    For the purposes of open delegation, READs and WRITEs done without an
 >    OPEN are treated as the functional equivalents of a corresponding
 >    type of OPEN.  This refers to the READs and WRITEs that use the
 >    anonymous stateid.  Therefore, READs or WRITEs with an anonymous
 >    stateid done by another client will force the server to recall a
 >    OPEN_DELEGATE_WRITE delegation.  A WRITE with an anonymous stateid
 >    done by another client will force a recall of OPEN_DELEGATE_READ
 >    delegations.

Did you mean to take out the 'all 1's case' which was an alternative in v33?

s9.10, para 1:
> to use a stateid of all 0's or all 1's
Ought to be changed to "anonymous" and "READ bypass"

s9.11, last para; s15.18.5, 5th from last para; s15.20.4, para 1: 
Reference s9.1.3 about incrementing seqid. (Three places all told.)

Editorial:
==========

s5.4, 2nd bullet labelled '1.': s/common to all object within given file 
system./common to all objects within the given file system./
[Very nitty: Could use a different label type for the second set of 
bullets.]

s6.2: s/RECOMMENED/RECOMMENDED/

s8.8, para 1: s/an zero-length string/a zero-length string/ [Missed it 
last time].

s9: s/has a secondary purpose/have a secondary purpose/

New s9.1.3 looks good -- except for a bit of finger trouble
s9.1.3, para 4: s/cuurent/current/
s9.1.3, para 5: s/raesonably/reasonably/
s9.1.3, para 6: s/wraparoundby/wraparound by/, s/gretater/greater/

s9.1.6, para 2: s/consiting/consisting/

s9.1.7, para 1: s/Subseuent/Subsequent/

s10.3.1, first bullet: s/(such as an OPENs specifying DENY=NONE)/(such 
as for [or at or on] OPENs specifying DENY=NONE)/

s12.4, 1st bullet: s/has a non-UT8 encoded/has a non-UTF8 encoded/
                                ^^^
s15.2.5, para 2: s/in which operation that/in which operations that/

s15.11.5, para 5: s/the server may return the error NFS4ERR_NOTSUPP./the 
server MAY return the error NFS4ERR_NOTSUPP./ (I think)

s15.18.6, last para: s/tfor the same client/for the same client/,
             s/i size and change attribute/size and change attributes/
               ^^^                     ^^^

s15.26.4/s15.26.5:  Not convinced about discussion on directory info 
changing under the feet of readdir.

s17, para 7: s/match wth the previous/match with the previous/
                      ^^^

On 18/11/14 15:17, Tom Haynes wrote:
>
>> On Nov 18, 2014, at 1:20 AM, Elwyn Davies <[email-masked]> wrote:
>>
>> Hi.
>>
>> Could you send me a snapshot of the updated version?  I am getting a bit lost on the results of the various changes.
>>
>> Cheers,
>> Elwyn
>>
>>
>
>
> Hi,
>
> I was wondering about that. :-)
>
> Later,
> Tom
>
>
>
>
>
>

---

From: David Noveck <[email-masked]>
Date: Mon, 1 Dec 2014 13:50:58 -0500

My view (remains) that if the cookie is not tied to the change attribute of
the directory, this makes life difficult for both the server (what does it
send if the data has been changed and how does it manage maintaining
checkpoints -- where a readdir finished --, in particular if the checkpoint
was on an entry that has been deleted), and the client.

The cookie is not tied to the change attribute of the directory.

The purpose of the cookie is ensuring that you do a READDIR that
encompasses multiple requests:

   - entries that exist during the entire period of the multi-requuest
   directory read appear exactly once in the result.
   - entries that exists during some part of the period of the
   multi-request directory read  appear at most once in the result.

Basically implementations scan through the directory as it exists on disk
and if there is a created entry, the client may or may not see it,
depending on where the the new entry happens to be put, with respect to the
ongoing scan.  As far as deleted entries, the implementation only has to be
able to move on to the next entry if it encouneters an entry marked as
deleted.

>    done by another client will force a recall of OPEN_DELEGATE_READ

suggest rewriting this as;

OPEN (anonymous and read-bypass stateids) are treated as the functional
equivalents of a corresponding type of OPEN.  READs and WRITEs done with an
anonymous stateid done by another client will force the server to recall a

 delegations.  Handling of read-bypass stateid's is identical except that a
READ

done with a read-bypass stateids will not force a recall of an
OPEN_DELEGATE_READ

delegation.

On Mon, Dec 1, 2014 at 9:41 AM, Elwyn Davies <[email-masked]> wrote:

> The one problem that I have left is the issue of directory contents being
> changed under the feet of a multi-part readdir request.  The last mail on
> this subject from 14 November said:
>
>>
>>
>>> >>> directory.  Presumably this is what is intended - it would certainly
> mean
>
>>> Asking an expertâ€¦
> of the directory, this makes life difficult for both the server (what does
> it send if the data has been changed and how does it manage maintaining
> checkpoints -- where a readdir finished --, in particular if the checkpoint
> was on an entry that has been deleted), and the client.
>
> Thoughts?
>
>
> s10.3.1, first bullet: s/(such as an OPENs specifying DENY=NONE)/(such as
> for [or at or on] OPENs specifying DENY=NONE)/
>
>>> Could you send me a snapshot of the updated version?  I am getting a bit
>>> lost on the results of the various changes.
>>>
>>> Cheers,
>>> Elwyn
>>>
>>>
>>>
>>
>> Hi,
>>

---

From: David Noveck <[email-masked]>
Date: Mon, 1 Dec 2014 14:44:22 -0500

We could substitute the first sentence of the paragraph as it appeared in
-34.

---

From: Elwyn Davies <[email-masked]>
Date: Mon, 01 Dec 2014 19:54:04 +0000

Hi, David.

Thanks for the response.

A couple of comments below...

Regards,
Elwyn

On 01/12/14 18:50, David Noveck wrote:
>     My view (remains) that if the cookie is not tied to the change
>     attribute of the directory, this makes life difficult for both the
>     server (what does it send if the data has been changed and how does
>     it manage maintaining checkpoints -- where a readdir finished --, in
>     particular if the checkpoint was on an entry that has been deleted),
>     and the client.
>
>
>   * entries that exist during the entire period of the multi-requuest
>   * entries that exists during some part of the period of the
> Basically implementations scan through the directory as it exists on
> disk and if there is a created entry, the client may or may not see it,
> depending on where the the new entry happens to be put, with respect to
> the ongoing scan.  As far as deleted entries, the implementation only
> has to be able to move on to the next entry if it encouneters an entry
> marked as deleted.

OK, so the server has to be clever rather than telling the client to 
restart the readdir -- I'm not sure this is the right trade-off since 
the client has to cope with being told the cached cookie is useless (as 
per Trond's input below) but I guess implementations work this way 
already and it isn't my call.  In that case I suggest that the last 
sentence of s15.26.4, para 9, is modified to tell server implementers a 
bit more clearly what they have to do.  Suggestion:
OLD:
    Ideally, the cookie value should not change if the directory is
    modified since the client may be caching these values.
NEW:
    The server SHOULD try to accept cookie values issued with READDIR
    responses even if the directory has been modified between the
    READDIR calls but MAY return NFS4ERR_NOT_VALID if this is not
    possible as might be the case if the server has rebooted in the
    interim.

One nasty thought occurs to me regarding the idea that a client caches
the cookie value(s):  Is the server supposed to maintain the whole
sequence of cookies for a sequence of READDIRs on a single directory so
the client can reissue some of the READDIR calls or is it fair enough 
for the server to throw them away after the client has used them once? 
And how long should the server hang onto cookie state in normal operation?

>       delegations.  Handling of read-bypass stateid's is identical
>     except that a READ
>
That sounds good.  It might be useful to add (e.g., to the end of para 1 
of s9.6.1:
"referred to as anonymous stateid and read-bypass stateid respectively."

>
>
>
> On Mon, Dec 1, 2014 at 9:41 AM, Elwyn Davies <[email-masked]
>     This looks pretty good to me and has sorted out almost everything
>     that I flagged up in the review. In particular s7.8 is much improved.
>
>     being changed under the feet of a multi-part readdir request.  The
>     last mail on this subject from 14 November said:
>
>
>
>             On Nov 14, 2014, at 3:05 PM, Tom Haynes
>             <[email-masked]
>             <mailto:[email-masked]>__> wrote:
>
>
>     <<snip>>
>
>                         <mailto:[email-masked]>
>                         <mailto:[email-masked]
>                         <mailto:[email-masked]>>__> wrote:
>
>
>
>
>     >>> directory.  Presumably this is what is intended - it would certainly mean
>
>     that changes would be picked up and READDIR would have to start from
>     the beginning again.
>
>
>
>             Asking an expertâ€¦
>
> 15.26.4
>          From Trond:
>
>     My view (remains) that if the cookie is not tied to the change
>     attribute of the directory, this makes life difficult for both the
>     server (what does it send if the data has been changed and how does
>     it manage maintaining checkpoints -- where a readdir finished --, in
>     particular if the checkpoint was on an entry that has been deleted),
>     and the client.
>
>     Thoughts?
>
>
>     I also found a some minor typos (mostly finger trouble in the new
>     bits) and there are a couple of nits outstanding (see below).
>
>     s8.5, last para: I am not sure that the jargon 'trunking relation'
>     is widely understood. [I don't think I have ever come across it.]
>
>     s19.4.1, para 2:
>     s5.4, 2nd bullet labelled '1.': s/common to all object within given
>     file system./common to all objects within the given file system./
>     s8.8, para 1: s/an zero-length string/a zero-length string/ [Missed
>     it last time].
>
>     s10.3.1, first bullet: s/(such as an OPENs specifying
>     DENY=NONE)/(such as for [or at or on] OPENs specifying DENY=NONE)/
>
>     s15.11.5, para 5: s/the server may return the error
>     NFS4ERR_NOTSUPP./the server MAY return the error NFS4ERR_NOTSUPP./
>     (I think)
>
>             On Nov 18, 2014, at 1:20 AM, Elwyn Davies
>             <[email-masked] <mailto:[email-masked]>> wrote:
>
>             Hi.
>
>             Could you send me a snapshot of the updated version?  I am
>             getting a bit lost on the results of the various changes.
>
>             Cheers,
>             Elwyn
>
>
>
>
>         Hi,
>

---

From: Tom Haynes <[email-masked]>
Date: Mon, 1 Dec 2014 13:30:45 -0800

> On Dec 1, 2014, at 6:41 AM, Elwyn Davies <[email-masked]> wrote:
> 
> Hi, Tom and David.
> 
> This looks pretty good to me and has sorted out almost everything that I flagged up in the review. In particular s7.8 is much improved.
> 
> The one problem that I have left is the issue of directory contents being changed under the feet of a multi-part readdir request.  The last mail on this subject from 14 November said:
>> 
>> 
> that changes would be picked up and READDIR would have to start from the beginning again.
>>>>> 
>>> 
>>> Asking an expertâ€¦
>> 
>> From Trond:
>> 
> My view (remains) that if the cookie is not tied to the change attribute of the directory, this makes life difficult for both the server (what does it send if the data has been changed and how does it manage maintaining checkpoints -- where a readdir finished --, in particular if the checkpoint was on an entry that has been deleted), and the client.
> 
> Thoughts?
> 
> 
> I also found a some minor typos (mostly finger trouble in the new bits) and there are a couple of nits outstanding (see below).
> 

Fixed

> 
> s8.5, last para: I am not sure that the jargon 'trunking relation' is widely understood. [I don't think I have ever come across it.]

This is RFC 5661 jargon.

New text:

   If a single location entry designates multiple server IP addresses,
   the client should choose a single one to use.  When two server
   addresses are designated by a single location entry and they
   correspond to different servers, this normally indicates some sort of
   misconfiguration, and so the client should avoid using such location
   entries when alternatives are available.  When they are not, clients
   should pick one of IP addresses and use it, without using others that
   are not directed to the same server.

> Did you mean to take out the 'all 1's case' which was an alternative in v33?

Dave, was this your editorial change?

> Ought to be changed to "anonymous" and "READ bypass”

I kinda get the first change:

      Even if the
      client intends to use an anonymous stateid, it must
      still obtain the filehandle for the regular file with the OPEN
      operation so the appropriate share semantics can be applied.

I don’t follow the second request...

> 
> s9.11, last para; s15.18.5, 5th from last para; s15.20.4, para 1: Reference s9.1.3 about incrementing seqid. (Three places all told.)

Done

> 
> Editorial:
> ==========
> 
> s5.4, 2nd bullet labelled '1.': s/common to all object within given file system./common to all objects within the given file system./
> [Very nitty: Could use a different label type for the second set of bullets.]
> 

done

> 
> s8.8, para 1: s/an zero-length string/a zero-length string/ [Missed it last time].

done

> s9: s/has a secondary purpose/have a secondary purpose/

done

> s9.1.7, para 1: s/Subseuent/Subsequent/

done

> 
> s10.3.1, first bullet: s/(such as an OPENs specifying DENY=NONE)/(such as for [or at or on] OPENs specifying DENY=NONE)/

at

> s12.4, 1st bullet: s/has a non-UT8 encoded/has a non-UTF8 encoded/

done

> s15.2.5, para 2: s/in which operation that/in which operations that/

ok

> 
> s15.11.5, para 5: s/the server may return the error NFS4ERR_NOTSUPP./the server MAY return the error NFS4ERR_NOTSUPP./ (I think)

ok

>            s/i size and change attribute/size and change attributes/

ok

> s15.26.4/s15.26.5:  Not convinced about discussion on directory info changing under the feet of readdir.

I think Dave is responding to this...

> s17, para 7: s/match wth the previous/match with the previous/

Caught some whch as well. :-)

---

From: Tom Haynes <[email-masked]>
Date: Mon, 1 Dec 2014 14:02:01 -0800

> On Dec 1, 2014, at 1:30 PM, Tom Haynes <[email-masked]> wrote:
> 
>> 

Okay, I understand the request based on the other mail thread.

---

From: Tom Haynes <[email-masked]>
Date: Mon, 1 Dec 2014 14:13:29 -0800

> On Dec 1, 2014, at 11:54 AM, Elwyn Davies <[email-masked]> wrote:
> 
> Hi, David.
> 
> Thanks for the response.
> 
> OK, so the server has to be clever rather than telling the client to restart the readdir -- I'm not sure this is the right trade-off since the client has to cope with being told the cached cookie is useless (as per Trond's input below) but I guess implementations work this way already and it isn't my call.  In that case I suggest that the last sentence of s15.26.4, para 9, is modified to tell server implementers a bit more clearly what they have to do.  Suggestion:
> OLD:
> the client can reissue some of the READDIR calls or is it fair enough for the server to throw them away after the client has used them once? And how long should the server hang onto cookie state in normal operation?
> 
>> 
>>    s19.4.1, para 2:
>> 

Done

> That sounds good.  It might be useful to add (e.g., to the end of para 1 of s9.6.1:
> "referred to as anonymous stateid and read-bypass stateid respectively.”

Willing to do, just not sure where you want this:

9.6.1.  Client Failure and Recovery

   In the event that a client fails, the server may recover the client's
   locks when the associated leases have expired.  Conflicting locks
   from another client may only be granted after this lease expiration.
   If the client is able to restart or reinitialize within the lease
   period the client may be forced to wait the remainder of the lease
   period before obtaining new locks.

I don’t think that line fits here...

---

From: Tom Haynes <[email-masked]>
Date: Mon, 1 Dec 2014 16:56:25 -0800

>   possible as might be the case if the server has rebooted in the

I’ve taken this change.

> the client can reissue some of the READDIR calls or is it fair enough for the server to throw them away after the client has used them once? And how long should the server hang onto cookie state in…

No, it does not have to save them.

But the cookies are pretty persistent in the first place. I.e., use some property of either the inode or the dirent to generate the cookie.

As such, the cookie state does not need to be held onto.

And for the life, it actually needs to be “stable” between the opendir() and closedir() call on the client. While we are safe on a client reboot, a client could end up holding onto the DIR pointer for a large indeterminate time.

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 02 Dec 2014 01:16:14 +0000

Hi.

I think we are just about done with this, unless you feel your last 
couple of statements (about stability etc) down the bottom could be 
usefully fed into the document.

I have a feeling that some of this is a bit reliant on the usual 
properties of Unix directories and could be a pig to implement on other 
filesystems - I don't remember enough about directories in NTFS etc (if 
I ever knew) to know whether it would work easily there.  However, since 
this has all been implemented, I'll just go quietly into the night.....

Thanks for your patience!

Please let me have a snapshot of the finished item if you don't submit a 
new version before the IESG meeting.

And now it must be bed time here in the UK!

Good might.
Elwyn

---

From: Tom Haynes <[email-masked]>
Date: Mon, 1 Dec 2014 17:28:05 -0800

Hi Elwyn,

Not sure if I’m allowed to post a new copy. Martin?

This has all of the changes accepted today.

Thanks,
Tom

---

From: Martin Stiemerling <[email-masked]>
Date: Tue, 2 Dec 2014 09:06:46 +0100

Hi Tom,

please wait until the telechat is over on Thurdsday. It starts at 11:30 ET und runs for approx 2hrs. 

Thanks!

  Martin

> On 02.12.2014, at 02:28, Tom Haynes <[email-masked]> wrote:
> 
> Hi Elwyn,
> 
> <draft-ietf-nfsv4-rfc3530bis-35.txt>

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 02 Dec 2014 14:00:41 +0000

Hi.

Sorry to keep harping on about READDIR, but...

Tom's comment about opendir/closedir at the end of the mail nbelow 
obviously triggered something in my brain overnight, 'cos I woke up 
realizing that I wasn't correctly understanding the semantics here.

Please bear with while I parade my naivete :-(

Having refreshed my memory on the semantics of reading directories in 
Unix and checked back on NFSv4, I realized that NFSv4 doesn't have the 
equivalents of opendir and closedir.

Hence, NFS has no equivalent of the "directory stream" referred to in 
the Unix manual pages.  Furthermore you don't 'open' a directory in 
NFSv4, you just give it the filehandle.

So, the case of the directory getting deleted under your feet between 
READDIR calls presumably just results in a NFS4ERR_STALE error on the 
next READDIR call and the issue of whether the cookie is valid is moot. 
It is then up to the client to map this into local OS behaviour.

This means that only the directory modification case is relevant to the 
cookie validity issue, and I understand the comment that David was 
making yesterday more clearly now.

However the lack of a closedir equivalent means that the server has no 
idea when the client has finished with the READDIR cookie(s).  I think 
it would be helpful to give some advice on what the server should do to 
avoid a build up of probably useless state.

Given the usual usage of READDIR, I would expect that once a call has 
returned eof<-TRUE, the client will in almost all cases be done and the 
state for the (client,directory) pair can be discarded.  However, there 
might be cases where the call has to be rerun due to comms failure I 
suppose, so maybe after a number of calls at eof?  Presumably there also 
ought to be some sort of timeout (like if the leases expire??)  I have 
no idea what current implementations do here.

Crafting a couple of sentences to mention these three items (deletion, 
modification and stale state) would IMO be helpful.  It would certainly 
have helped me!

Regards,
Elwyn

---

From: David Noveck <[email-masked]>
Date: Tue, 2 Dec 2014 09:56:39 -0500

Hence, NFS has no equivalent of the "directory stream" referred to in the
Unix manual pages.  Furthermore you don't  'open' a directory in NFSv4, you
just give it the filehandle.

Correct.  NFSv4 just took over the NFSv3 approach to reading directories
and NFSv3 doesn't support open/close at all.

However the lack of a closedir equivalent means that the server has no idea
when the client has finished with the READDIR cookie(s).

Right.

 I think it would be helpful to give some advice on what the server should
do to avoid a build up of probably useless state.

how about:

As there is no way for the client to indicate that a cookie value once
received, will not be subsequently used, serer implementations should avoid
schemes that allocate memory corresponding to a returned cookie.  Such
allocation can be avoided if the server bases cookie values on a value such
as the offset within the directory wher the scan is to be resumed.

Given the usual usage of READDIR, I would expect that once a call has
returned eof<-TRUE, the client will in almost all cases be done and the
state for the (client,directory) pair can be discarded.

Except there is no state for  client,directory pair.  The spec does not
mandate that such state be maintained, and i don't know of server that
maintain that sort of state.  NFSv4. servers follow the NFSv3 practice,
where client state* per se* did not exist.

However, there might be cases where the call has to be rerun due to comms
failure I suppose, so maybe after a number of calls at eof?  Presumably
there also ought to be some sort of timeout (like if the leases expire??)
I have no idea what current implementations do here.

None of these situations arise since there is no per--client readdir state.

have helped me!.

suggestions:

   - *deletion*

cookies return by READDIR become invalid when the directory is removed but
the server is not obliged for check for this situation or check the cookie
verifier since the directory handle will become stale.

   - *modification*

Cookies generated by such techniques should reaiin valid despite
modification of the associated directory.  If a server were to invalidate a
cookie because of a directory modification, READDIR's of large directories
might never finish.

   - *stale state*

I think it is clear from the text above that there is no state to become
stale.  Not sure what else you can say.

   - *directory reorganization*

certain re-organization operation on a directory (including directory
compaction) may invalidate READDDIR cookies previously given out.  when
these situation ocur, the server should modify the cookie verifier to
accepting cookies which are no longer valid.

On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies <[email-masked]> wrote:

> Having refreshed my memory on the semantics of reading directories in Unix
> and checked back on NFSv4, I realized that NFSv4 doesn't have the
> Hence, NFS has no equivalent of the "directory stream" referred to in the
> Unix manual pages.  Furthermore you don't 'open' a directory in NFSv4, you
> just give it the filehandle.
>
> READDIR calls presumably just results in a NFS4ERR_STALE error on the next
> READDIR call and the issue of whether the cookie is valid is moot. It is
> then up to the client to map this into local OS behaviour.
>
> cookie validity issue, and I understand the comment that David was making
> yesterday more clearly now.
>
> idea when the client has finished with the READDIR cookie(s).  I think it
> would be helpful to give some advice on what the server should do to avoid
> a build up of probably useless state.
>
> ought to be some sort of timeout (like if the leases expire??)  I have no
> idea what current implementations do here.
>
>>> restart the readdir -- I'm not sure this is the right trade-off since the
>>> client has to cope with being told the cached cookie is useless (as per
>>> Trond's input below) but I guess implementations work this way already and
>>> it isn't my call.  In that case I suggest that the last sentence of
>>> s15.26.4, para 9, is modified to tell server implementers a bit more
>>> clearly what they have to do.  Suggestion:
>>> OLD:
>>> for the server to throw them away after the client has used them once? And
>>> how long should the server hang onto cookie state in normal operation?
>>>
>>>
>> But the cookies are pretty persistent in the first place. I.e., use some
>> property of either the inode or the dirent to generate the cookie.
>>
>> And for the life, it actually needs to be “stable” between the opendir()
>> and closedir() call on the client. While we are safe on a client reboot, a
>> client could end up holding onto the DIR pointer for a large indeterminate
>> time.
>>
>>

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 2 Dec 2014 07:54:30 -0800

> On Dec 2, 2014, at 6:00 AM, Elwyn Davies <[email-masked]> wrote:
> 
> Hi.
> 
> Tom's comment about opendir/closedir at the end of the mail nbelow obviously triggered something in my brain overnight, 'cos I woke up realizing that I wasn't correctly understanding the semantics here.
> 
> Having refreshed my memory on the semantics of reading directories in Unix and checked back on NFSv4, I realized that NFSv4 doesn't have the equivalents of opendir and closedir.
> 

I was actually talking about on the client, aka opendir(3) and closedir(3) on Linux. So while the protocol may have no notion of these, these do exist and interoperate with the protocol.

In Linux, we have:

const struct file_operations nfs_dir_operations = {
        .llseek         = nfs_llseek_dir,
        .read           = generic_read_dir,
        .iterate        = nfs_readdir,
        .open           = nfs_opendir,
        .release        = nfs_closedir,
        .fsync          = nfs_fsync_dir,
};

I.e., the client is aware of when the closedir(3) occurs.

> Hence, NFS has no equivalent of the "directory stream" referred to in the Unix manual pages.  Furthermore you don't 'open' a directory in NFSv4, you just give it the filehandle.
> 
> So, the case of the directory getting deleted under your feet between READDIR calls presumably just results in a NFS4ERR_STALE error on the next READDIR call and the issue of whether the cookie is valid is moot. It is then up to the client to map this into local OS behaviour.
> 
> This means that only the directory modification case is relevant to the cookie validity issue, and I understand the comment that David was making yesterday more clearly now.
> 
> However the lack of a closedir equivalent means that the server has no idea when the client has finished with the READDIR cookie(s).  I think it would be helpful to give some advice on what the server should do to avoid a build up of probably useless state.
> 
> Given the usual usage of READDIR, I would expect that once a call has returned eof<-TRUE, the client will in almost all cases be done and the state for the (client,directory) pair can be discarded.  However, there might be cases where the call has to be rerun due to comms failure I suppose, so maybe after a number of calls at eof?  Presumably there also ought to be some sort of timeout (like if the leases expire??)  I have no idea what current implementations do here.

This is like any other call to the server which hangs - most likely an EDELAY is passed back up to the caller on the client.

> 
> Crafting a couple of sentences to mention these three items (deletion, modification and stale state) would IMO be helpful.  It would certainly have helped me!
> 
> Regards,
> Elwyn
>

---

From: Elwyn Davies <[email-masked]>
Date: Tue, 02 Dec 2014 16:41:19 +0000

Hi, David.

Excellent.  I think some suitable combination of your words below would 
fill the bill.

The point about the very large directories is well taken - I hadn't 
grokked that previously.

I guess the update should go in s15.26.5 after para 1:

Something like:

-----------------

As there is no way for the client to indicate that a cookie value once 
received, will not be subsequently used, server implementations should 
avoid schemes that allocate memory corresponding to a returned cookie. 
Such allocation can be avoided if the server bases cookie values on a 
value such as the offset within the directory where the scan is to be 
resumed.

Cookies generated by such techniques should be designed to remain valid 
despite modification of the associated directory.  If a server were to 
invalidate a cookie because of a directory modification, READDIR's of 
large directories might never finish.

If a directory is deleted after the client has carried out one or more 
READDIR operations on the directory, the cookies returned will become 
invalid but the server does not need to be concerned as the directory 
file handle used previously would have become stale and would be 
reported as such on subsequent READDIR operations.  The server would not 
need to check the cookie verifier in this case.

However, certain re-organization operations on a directory (including 
directory compaction) may invalidate READDDIR cookies previously given 
out.  When such a situation occurs, the server should modify the cookie 
verifier so as to accept cookies which would otherwise no longer be valid.

-------------------

Aside: handling reorganization without creating state sounds challenging 
but life wouldn't be interesting without challenges! :-)

A couple of nits that got missed from yesterday:

s9.1.3, last sentence: s/an can be treated as later./and can be treated 
as later./
                          ^^^^

With something like the above, I think we are done.

Congratulations!  I hope this gets to be an RFC RSN.

Cheers,
Elwyn

On 02/12/14 14:56, David Noveck wrote:
>     Hence, NFS has no equivalent of the "directory stream" referred to
>     in the Unix manual pages.  Furthermore you don't 'open' a directory
>     in NFSv4, you just give it the filehandle.
>
>
>     However the lack of a closedir equivalent means that the server has
>     no idea when the client has finished with the READDIR cookie(s).
>
> Right.
>
>       I think it would be helpful to give some advice on what the server
>     should do to avoid a build up of probably useless state.
>
> how about:
>
>     As there is no way for the client to indicate that a cookie value
>     once received, will not be subsequently used, serer implementations
>     should avoid schemes that allocate memory corresponding to a
>     returned cookie.  Such allocation can be avoided if the server bases
>     cookie values on a value such as the offset within the directory
>     wher the scan is to be resumed.
>
>
>     Given the usual usage of READDIR, I would expect that once a call
>     has returned eof<-TRUE, the client will in almost all cases be done
>     and the state for the (client,directory) pair can be discarded.
>
>
> where client state/per se/ did not exist.
>
>     However, there might be cases where the call has to be rerun due to
>     comms failure I suppose, so maybe after a number of calls at eof?
>     Presumably there also ought to be some sort of timeout (like if the
>     leases expire??)  I have no idea what current implementations do here.
>
>   * *deletion*
>     cookies return by READDIR become invalid when the directory is
>     removed but the server is not obliged for check for this situation
>     or check the cookie verifier since the directory handle will become
>     stale.
>
>   * *modification*
>
>     modification of the associated directory.  If a server were to
>     invalidate a cookie because of a directory modification, READDIR's
>     of large directories might never finish.
>
>   * *stale state*
>
>     I think it is clear from the text above that there is no state to
>     become stale.  Not sure what else you can say.
>
>   * *directory reorganization*
>
>     certain re-organization operation on a directory (including
>     directory compaction) may invalidate READDDIR cookies
>     previously given out.  when these situation ocur, the server should
>     modify the cookie verifier to accepting cookies which are no longer
>     valid.
>
>
> On Tue, Dec 2, 2014 at 9:00 AM, Elwyn Davies <[email-masked]
>     Having refreshed my memory on the semantics of reading directories
>     in Unix and checked back on NFSv4, I realized that NFSv4 doesn't
>     have the equivalents of opendir and closedir.
>
>     Hence, NFS has no equivalent of the "directory stream" referred to
>     in the Unix manual pages.  Furthermore you don't 'open' a directory
>     in NFSv4, you just give it the filehandle.
>
>     So, the case of the directory getting deleted under your feet
>     between READDIR calls presumably just results in a NFS4ERR_STALE
>     error on the next READDIR call and the issue of whether the cookie
>     is valid is moot. It is then up to the client to map this into local
>     OS behaviour.
>
>     This means that only the directory modification case is relevant to
>     the cookie validity issue, and I understand the comment that David
>     was making yesterday more clearly now.
>
>     However the lack of a closedir equivalent means that the server has
>     no idea when the client has finished with the READDIR cookie(s).  I
>     think it would be helpful to give some advice on what the server
>     should do to avoid a build up of probably useless state.
>
>     Given the usual usage of READDIR, I would expect that once a call
>     has returned eof<-TRUE, the client will in almost all cases be done
>     and the state for the (client,directory) pair can be discarded.
>     However, there might be cases where the call has to be rerun due to
>     comms failure I suppose, so maybe after a number of calls at eof?
>     Presumably there also ought to be some sort of timeout (like if the
>     leases expire??)  I have no idea what current implementations do here.
>
>     Crafting a couple of sentences to mention these three items
>     (deletion, modification and stale state) would IMO be helpful.  It
>     would certainly have helped me!
>
>     Regards,
>     Elwyn
>
>             On Dec 1, 2014, at 11:54 AM, Elwyn Davies
>             OK, so the server has to be clever rather than telling the
>             client to restart the readdir -- I'm not sure this is the
>             right trade-off since the client has to cope with being told
>             the cached cookie is useless (as per Trond's input below)
>             but I guess implementations work this way already and it
>             isn't my call.  In that case I suggest that the last
>             sentence of s15.26.4, para 9, is modified to tell server
>             implementers a bit more clearly what they have to do.
>             Suggestion:
>             OLD:
>             the client can reissue some of the READDIR calls or is it
>             fair enough for the server to throw them away after the
>             client has used them once? And how long should the server
>             hang onto cookie state in normal operation?
>
>
>         But the cookies are pretty persistent in the first place. I.e.,
>         use some property of either the inode or the dirent to generate
>         the cookie.
>
>         And for the life, it actually needs to be “stable” between the
>         opendir() and closedir() call on the client. While we are safe
>         on a client reboot, a client could end up holding onto the DIR
>         pointer for a large indeterminate time.
>
>

---

From: David Noveck <[email-masked]>
Date: Tue, 2 Dec 2014 12:34:29 -0500

> cookies previously given out.  When such a situation occurs, the server
should modify the cookie verifier so as to
>  accept cookies which would otherwise no longer be vali

''accept cookies'" -->  "disallow use of cookies'

On Tue, Dec 2, 2014 at 11:41 AM, Elwyn Davies <[email-masked]> wrote:

> avoid schemes that allocate memory corresponding to a returned cookie. Such
> as the offset within the directory where the scan is to be resumed.
>
> invalidate a cookie because of a directory modification, READDIR's of large
> directories might never finish.
>
> invalid but the server does not need to be concerned as the directory file
> handle used previously would have become stale and would be reported as
> such on subsequent READDIR operations.  The server would not need to check
> the cookie verifier in this case.
>
> s9.1.3, last sentence: s/an can be treated as later./and can be treated as
> later./
>                          ^^^^
>
>

---

From: Tom Haynes <[email-masked]>
Date: Tue, 2 Dec 2014 10:15:40 -0800

Hi Elwyn,

Thanks for all of the review and help effort. Besides the inexplicable typos, you unearthed some issues which would give an experienced reader pause.

Tom

> On Dec 2, 2014, at 8:41 AM, Elwyn Davies <[email-masked]> wrote:
> 
> Hi, David.
> 
> Excellent.  I think some suitable combination of your words below would fill the bill.
> 
> The point about the very large directories is well taken - I hadn't grokked that previously.
> 
> As there is no way for the client to indicate that a cookie value once received, will not be subsequently used, server implementations should avoid schemes that allocate memory corresponding to a returned cookie. Such allocation can be avoided if the server bases cookie values on a value such as the offset within the directory where the scan is to be resumed.
> 
> Cookies generated by such techniques should be designed to remain valid despite modification of the associated directory.  If a server were to invalidate a cookie because of a directory modification, READDIR's of large directories might never finish.
> 
> If a directory is deleted after the client has carried out one or more READDIR operations on the directory, the cookies returned will become invalid but the server does not need to be concerned as the directory file handle used previously would have become stale and would be reported as such on subsequent READDIR operations.  The server would not need to check the cookie verifier in this case.
> 
> However, certain re-organization operations on a directory (including directory compaction) may invalidate READDDIR cookies previously given out.  When such a situation occurs, the server should modify the cookie verifier so as to accept cookies which would otherwise no longer be valid.
> 

Done, with Dave’s suggested tweak.

> -------------------
> 
> Aside: handling reorganization without creating state sounds challenging but life wouldn't be interesting without challenges! :-)
> 

Done

> 
> s9.1.3, last sentence: s/an can be treated as later./and can be treated as later./
>                         ^^^^
> 

Done