Subject: Re: [nfsv4] Scribe 6/28

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 2 Jul 2012 16:02:31 +0000

On Jun 29, 2012, at 7:27 PM, Dean wrote:

> 
>>> For example with forthcoming writes, the server may decide to bypass its
>>> cache and
>>> use direct I/O to place the data directly on disk.  This may allow the
>>> server
>>> to avoid polluting its cache and keep data in the cache that has a
>>> greater possibly of re-access.
>>> Another example is with forthcoming read requests,  where the server may
>>> decide to either use direct I/O, avoiding
>>> its cache altogether, or use a separate cache that will avoid evicting data
>>> that has a greater possibly of re-access.
>> I don't see any value add here either.  And the text certainly belongs in
>> the IMPLEMENTATION (13.8.4) notes.
>> 
>> >From what I can tell, your use case is a client extending O_DIRECT to
>> the caching policies on the server. Nothing in the draft prevents this
>> from occurring. But I heard others yesterday wanting
>> to preserve the ability for O_DIRECT to not dictate what happens on the server.
>> 
>> And that choice is in the API on the client, not in the protocol.
> 
> I think you have misread my statement.  This has nothing to do with O_DIRECT.

Given your past terminology, direct I/O == O_DIRECT.

   Direct I/O can be used in Linux and AIX via the open()
   O_DIRECT parameter, in Solaris via the directio() function, and in
   Windows via the CreateFile() FILE_FLAG_NO_BUFFERING flag.

> 
> This text just gives motivation for why we want to support posix_fadvise NOREUSE in the NFS protocol.

Why do we need to provide motivation to do something we have already decided to do?

> Dean
> 
>> 
>> 
>> 
>>> 
>>> Second, I propose to remove the following:
>>>   1498       Another use case is "Direct I/O", which allows a stated
>>> holder to
>>>   1499       inform the server that it does not wish to cache data.
>>> Today, for
>>>   1500       applications that only intend to read data once, the use
>>> of direct
>>>   1501       I/O disables client caching, but does not affect server
>>> caching.  By
>>>   1502       caching data that will not be re-read, the server is
>>> polluting its
>>>   1503       cache and possibly causing useful cached data to be
>>> evicted.  By
>>>   1504       informing the server of its expected I/O access, this
>>> situation can
>>>   1505       be avoid.  Direct I/O can be used in Linux and AIX via the
>>> open()
>>>   1506       O_DIRECT parameter, in Solaris via the directio()
>>> function, and in
>>>   1507       Windows via the CreateFile() FILE_FLAG_NO_BUFFERING flag.
>>> 
>>> Third, I propose to remove the following text:
>>>   4060       Another useful point would be when an application
>>> indicates it is
>>>   4061       using direct I/O. Direct I/O may be specified at file
>>> open, in which
>>>   4062       case a IO_ADVISE may be included in the same compound as
>>> the OPEN
>>>   4063       operation with the IO_ADVISE4_NOREUSE flag set.  Direct
>>> I/O may also
>>>   4064       be specified separately, in which case a IO_ADVISE
>>> operation can be
>>>   4065       sent to the server separately.  As above, IO_ADVISE4_WRITE and
>>>   4066       IO_ADVISE4_READ may be set based upon the type of file access
>>>   4067       specified when the file was opened.
>>> 
>>> Fourth, I propose to change the following text:
>>>   4055       The most obvious is in direct response to an application's
>>> execution
>>>   4056       of posix_fadvise.  In this case, IO_ADVISE4_WRITE and
>>> IO_ADVISE4_READ
>>>   4057       may be set based upon the type of file access specified
>>> when the file
>>>   4058       was opened.
>>> to
>>> The most obvious is in direct response to an application's execution
>>> of posix_fadvise.  The arguments from posix_fadvise would be translated
>>> directly to IO_ADVISE and sent immediately to the server. The posix_fadvise
>>> hints specified in Section 3.2 translate to IO_ADVISE hints by replacing
>>> the prefix POSIX_FADV with IO_ADVISE4. In this case,
>>> IO_ADVISE4_WRITE and IO_ADVISE4_READ
>>> may be set based upon the type of file access specified when the file
>>> was opened.
>>> 
>>> Fifth, while not required, I noticed that section 3.3 doesn't have any
>>> motivational text for IO_ADVISE4_INIT_PROXIMITY.  Just a thought, but
>>> might be worth considering.
>>> 
>>> Dean
>>>> 3905 - Trond and Chuck: What is the STATE_ ID, is it limited? Open
>>>> block or delegation
>>>> 
>>>> Dean: The lifetime of a file hint is the open and the lifetime of byte
>>>> range hint is a compound
>>>> 
>>>> Chuck: Wants an explicit argument to state what the lifetime is
>>>> 
>>>> **
>>>> 
>>>> *END *
>>>> 
>>>> *Begin next meeting at: 4069*
>>>> 
>>>> Cheers!
>>>> 
>>>> *Talia Reyes-Ortiz*
>>>> Program Manager
>>>> CTO Office
>>>> 
>>>> *NetApp Inc.*
>>>> 408.822.1566 Direct Phone
>>>> [email-masked]
>>>> 
>>>> Description:
>>>> http://media.netapp.com/images/esigs-innovative-company-480x75.png
>>>> <http://www.forbes.com/special-features/innovative-companies.html>
>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> nfsv4 mailing list
>>>> [email-masked]
>>>> https://www.ietf.org/mailman/listinfo/nfsv4
>>> 
>>> _______________________________________________
>>> nfsv4 mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/nfsv4
>> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 2 Jul 2012 16:21:36 +0000

On Jun 29, 2012, at 7:22 PM, Dean wrote:

On 6/29/12 2:28 PM, [email-masked]<mailto:[email-masked]> wrote:
Good point.  It could and should be removed.  So how about:

POSIX_FADV_NOREUSE -  This hint specifies that the application expects to
access the specified data once and then not reuse it thereafter.
The intention is to allow the server to improve its caching policy regarding the specified data.
For example, with forthcoming reads and writes, the server may decide to make the data read or written
more likely to be selected for cache reuse than it would be otherwise. This may allow the server
to keep data in its cache that has a higher probability of re-access.

If people still want more succinctness, we could go with what is currently in the document.

>From the call, it seems there is some confusion on how posix_fadvise would map to IO_ADVISE, so I thought this would help.

The confusion on the call had nothing to do with NOREUSE and everything to do with when the IO_ADVISE would be sent
and what the lifetime would be for that hint. I had thought the issue was that the current draft was lacking in describing whether
the hint applied to the next read or write OR the life of the compound.

What I heard Trond state was that every hint only lived within a compound. I may have that wrong, so Trond could you please
state what your issue is here?

In any event, the issue is not motivational, it is in the implementation. We are all motivated to get IO_ADVISE into NFSv4.2.  If there is a major disagreement on implementation, then there is a risk of it being derailed from the release.

One thing I tried to capture is that this hint acts on forthcoming I/O, and is not intended as a hint regarding past I/O (that is what DONTNEED is for).   So one thing we could do is be more clear about this in the IO_ADVISE section instead.

Dean

From: Peter Staubach [mailto:[email-masked]]
Sent: Friday, June 29, 2012 5:17 PM
To: Noveck, David; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: [nfsv4] Scribe 6/28

I am not sure that I see the value in comparing NFSv4.X to previous versions of NFS.  Does that sentence really add value or could it be removed and keep the paragraph more succinct?

                Thanx…

                                ps

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Friday, June 29, 2012 5:12 PM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [nfsv4] Scribe 6/28

I think that what is there now is fine but if you want some elaboration, I would prefer something more like the following.

POSIX_FADV_NOREUSE -  Specifies that the application expects to
access the specified data once and then not reuse it thereafter.
In previous NFS versions, the NFS client had no way of passing information about such
application expectations to the server. This hint allows the application to specify its expectations about
forthcoming data use and so allow the server to improve its caching policy regarding the specified data.
For example, with forthcoming reads and writes, the server may decide to make the data read or written
more likely to be selected for cache reuse than it would be otherwise. This may allow the server
to keep data in its cache that has a higher probability of re-access.

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]<mailto:[mailto:[email-masked]]> On Behalf Of Dean
Sent: Friday, June 29, 2012 4:45 PM
To: [email-masked]<mailto:[email-masked]>; Eisler, Michael
Subject: Re: [nfsv4] Scribe 6/28

On 6/28/12 4:05 PM, Reyes-Ortiz, Talia wrote:
Chuck, Tom, Bryan, Trond, Talia, Dean

Note Well Agreement. See: http://www.ietf.org/NOTEWELL.html

START

Dean: Going to remove O_DIRECT to give a specific example for IO_ADVISE_NOREUSE

First, I propose to change the following text:
  1466       POSIX_FADV_NOREUSE -  Specifies that the application expects to
  1467          access the specified data once and then not reuse it thereafter.
to
POSIX_FADV_NOREUSE -  Specifies that the application expects to
access the specified data once and then not reuse it thereafter.
Currently, the NFS client has no way of passing caching hints to the server for
forthcoming I/O requests.  This hint allows the application to specify data access information
that will allow the server to optimize its caching policy regarding the specified data.
For example with forthcoming writes, the server may decide to bypass its cache and
use direct I/O to place the data directly on disk.  This may allow the server
to avoid polluting its cache and keep data in the cache that has a greater possibly of re-access.
Another example is with forthcoming read requests,  where the server may decide to either use direct I/O, avoiding
its cache altogether, or use a separate cache that will avoid evicting data
that has a greater possibly of re-access.

Second, I propose to remove the following:
  1498       Another use case is "Direct I/O", which allows a stated holder to
  1499       inform the server that it does not wish to cache data.  Today, for
  1500       applications that only intend to read data once, the use of direct
  1501       I/O disables client caching, but does not affect server caching.  By
  1502       caching data that will not be re-read, the server is polluting its
  1503       cache and possibly causing useful cached data to be evicted.  By
  1504       informing the server of its expected I/O access, this situation can
  1505       be avoid.  Direct I/O can be used in Linux and AIX via the open()
  1506       O_DIRECT parameter, in Solaris via the directio() function, and in
  1507       Windows via the CreateFile() FILE_FLAG_NO_BUFFERING flag.

Third, I propose to remove the following text:
  4060       Another useful point would be when an application indicates it is
  4061       using direct I/O. Direct I/O may be specified at file open, in which
  4062       case a IO_ADVISE may be included in the same compound as the OPEN
  4063       operation with the IO_ADVISE4_NOREUSE flag set.  Direct I/O may also
  4064       be specified separately, in which case a IO_ADVISE operation can be
  4065       sent to the server separately.  As above, IO_ADVISE4_WRITE and
  4066       IO_ADVISE4_READ may be set based upon the type of file access
  4067       specified when the file was opened.

Fourth, I propose to change the following text:
  4055       The most obvious is in direct response to an application's execution
  4056       of posix_fadvise.  In this case, IO_ADVISE4_WRITE and IO_ADVISE4_READ
  4057       may be set based upon the type of file access specified when the file
  4058       was opened.
to
The most obvious is in direct response to an application's execution
of posix_fadvise.  The arguments from posix_fadvise would be translated
directly to IO_ADVISE and sent immediately to the server. The posix_fadvise
hints specified in Section 3.2 translate to IO_ADVISE hints by replacing
the prefix POSIX_FADV with IO_ADVISE4. In this case,
IO_ADVISE4_WRITE and IO_ADVISE4_READ
may be set based upon the type of file access specified when the file
was opened.

Fifth, while not required, I noticed that section 3.3 doesn't have any motivational text for IO_ADVISE4_INIT_PROXIMITY.  Just a thought, but might be worth considering.

Dean

3905 - Trond and Chuck: What is the STATE_ ID, is it limited? Open block or delegation

Dean: The lifetime of a file hint is the open and the lifetime of byte range hint is a compound

END
Begin next meeting at: 4069

Cheers!
Talia Reyes-Ortiz
Program Manager
CTO Office
NetApp Inc.
408.822.1566 Direct Phone
[email-masked]<mailto:[email-masked]>
<Mail Attachment.png><http://www.forbes.com/special-features/innovative-companies.html>

nfsv4 mailing list

[email-masked]<mailto:[email-masked]>

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Chuck Lever <[email-masked]>
Date: Mon, 2 Jul 2012 13:10:01 -0400

Hi-

>   Windows via the CreateFile() FILE_FLAG_NO_BUFFERING flag.

It seems to me that mentioning direct I/O even on the server is confusing to even expert readers, especially since there does not seem to be a universal and standard definition for what direct I/O is and why an application might choose to use it.

I further agree that there does not seem to be a specific compelling reason to bring up direct I/O here.  I suggest, for clarity, that we not mention direct I/O anywhere in the discussion of IO_ADVISE.

> Why do we need to provide motivation to do something we have already decided to do?

I think some justification for adding protocol support for posix_fadvise, in general, would be valuable.  This could take the form of "here are a couple of different workloads that benefit when the server is privy to the same caching hints the client is: see benchmarks."  This does not have to be specific language, and I think we've already seen some appropriate text samples on the mailing list.

But NOREUSE itself comes right along with posix_fadvise.  Is there a reason that we need to specially justify it's inclusion?  I can't think of one.

I think specific rationale should be included that covers:

 o  The addition of INIT_PROXIMITY and any issues it might bring with it,

 o  A discussion of how state ids are to be used with this interface, and

 o  Hint lifetime alternatives

In other words, exactly what we have been discussing on the phone calls (minus the O_DIRECT thread).

I'll re-iterate what Tom said in another e-mail: we believe IOADVISE is valuable, and would like to see it in NFSv4.2.  Now let's try to come to some consensus about how it will work.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Mon, 2 Jul 2012 18:49:56 +0000

On Mon, 2012-07-02 at 16:21 +0000, Haynes, Tom wrote:
> > On 6/29/12 2:28 PM, [email-masked] wrote:
> > 
> > >From the call, it seems there is some confusion on how
> > posix_fadvise would map to IO_ADVISE, so I thought this would help.
> > 
> 
> 
> The confusion on the call had nothing to do with NOREUSE and
> everything to do with when the IO_ADVISE would be sent
> and what the lifetime would be for that hint. I had thought the issue
> was that the current draft was lacking in describing whether
> What I heard Trond state was that every hint only lived within a
> compound. I may have that wrong, so Trond could you please

No. I don't say that every hint works that way. I'm saying that a client
that runs multiple processes and applications may need to manage the
hints depending on what kind of I/O the different applications are
performing on the different parts of the file.

Here is a proposal for a solution that doesn't require the server to
maintain any additional state, and yet gives the client the flexibility
to manage the hints on a byte range:

The proposal is to drop the problematic IOADVISE_NOREUSE from the spec,
and instead allow NFS clients to emulate fadvise(POSIX_FADV_NOREUSE)
using IO_ADVISE_DONTNEED.
Clients can do that either by appending an IO_ADVISE(IO_ADVISE_DONTNEED)
operation to the end of each READ/WRITE compound (so that it covers the
range just read), or by calling it in a separate IO_ADVISE compound that
is sent after the application is finished reading the range covered by
the fadvise call.
The disadvantage is that it means that the server won't know that it
could have satisfied that READ using an O_DIRECT/zero copy read from
disk directly to the RPC reply buffer (instead of going through the
buffer/page cache), but maybe that isn't a concern for server vendors.

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 3 Jul 2012 14:30:47 +0000

>using IO_ADVISE_DONTNEED.

The Linux man page is interesting:

  In kernels before 2.6.18, POSIX_FADV_NOREUSE had the same semantics as
  POSIX_FADV_WILLNEED. This was probably a bug; since kernel 2.6.18, this
  flag is a no-op.

  POSIX_FADV_DONTNEED attempts to free cached pages associated with
  the specified region. This is useful, for example, while streaming
  large files. A program may periodically request the kernel to free
  cached data that has already been used, so that more useful cached
  pages are not discarded instead.

While this information is a bit dated, it will not be by much. The flag is
now used: http://lwn.net/Articles/480930/

Some other webisms:

  As of this writing (2.6.21) Linux does not remember POSIX_FADV_DONTNEED
  advice for an open file. It acts when the advice is given, and when it
can
  not comply it forgets the advice. So it is up to you to make sure Linux
can
  comply.

Going back to http://lwn.net/Articles/480930/ :

  The list of FADV_NOREUSE ranges are maintained into an interval tree [5]
inside
  the address_space structure (for this a generic interval tree
implementation is
  also provided by PATCH 1/2). The intervals are dropped when pages are
also
  dropped from the page cache or when a different caching hint is
specified on
  the same intervals.

After reading that, remind me again why we care to add restrictions as to
the lifetime or number of byte range hints that can be outstanding? We are
basically changing the behavior of posix_fadvise when we do that.

These examples point out that implementations of posix_fadvise change over
time.

My proposal is that we throw out the text which talks about lifetime and
point out that IO_ADVISE is just an api to the server's implementation of
either posix_fadvise and/or non-posix code. Whether the underlying code
supports a hint or a certain number of outstanding byte range hints, then
that all happens silently - just like it does when Linux made
POSIX_FADV_NOREUSE be a no-op.

>is sent after the application is finished reading the range covered by

Look at http://lwn.net/Articles/480930/ for why you don't want to do that,
at least for Linux.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 3 Jul 2012 11:13:52 -0400

Hi-

> POSIX_FADV_NOREUSE be a no-op.

I suspect server implementations will want to be simple and stateless.  That is, they will want a hint to apply during a single compound, and that's it.

NFS clients can keep track of what hint to send in which compound.  This would allow clients to implement either single-request hints or hints that last until the next posix_fadvise(2) call, by including the same hint in every READ/WRITE compound.  If the client's fadvise() semantics change, then it should be straightforward simply to change the pattern of hints being sent to the server, and might allow non-POSIX clients to create their own specialized cache hints.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 3 Jul 2012 18:53:38 +0000

On 7/3/12 10:13 AM, "Chuck Lever" <[email-masked]> wrote:

>I suspect server implementations will want to be simple and stateless.
>That is, they will want a hint to apply during a single compound, and
>that's it.

Do I detect a hidden agenda for TSM? :->

Given that the client has no method to revoke/reclaim that state on the
server, why do we care at all whether the server can only store 1 or N
hints? Why should we paint ourselves into a corner here?

Hints apply to the given byte ranges until they do not. That might be
because the server succumbed to memory back pressure, another client
provided another hint, or the server detected a better algorithm
heuristically.

It doesn't matter.

The server should be like any other storage layer used by the application
- it can provide hints, but it has no clue if the layer is doing anything.

>
>NFS clients can keep track of what hint to send in which compound.  This
>would allow clients to implement either single-request hints or hints
>that last until the next posix_fadvise(2) call, by including the same
>hint in every READ/WRITE compound.  If the client's fadvise() semantics
>change, then it should be straightforward simply to change the pattern of
>hints being sent to the server, and might allow non-POSIX clients to
>create their own specialized cache hints.

Smart client, dumb server. Sure, doesn't matter.

Might cause thrashing on the server...

---

From: <[email-masked]>
Date: Thu, 5 Jul 2012 11:43:43 -0400

> Do I detect a hidden agenda for TSM? :->

If there is one, it's hidden from me. :-)

My (not so) hidden agenda with regard to hints and TSM is that
I won't need to address this in a forthcoming version of
draft-ietf-nfsv4-migration-issues.  If there are migration
issues with regard to this, they need to be addressed in the v4.2 
spec.

So I think we have to choose one of two models to make it clear whose
responsibility this is:

1) It's the server's responsibility to transfer this information, and
it SHOULD do so, although given that these are hints there is no guarantee.
In any case, there is no point in the client re-hinting on the destination.

2) It's the client's responsibility to rehint on the destination and it SHOULD 
do so  Given that, there is no expectation that hint information will move as 
part of migration.

Any compromise/avoidance is worse than either (1) or (2).

If we leave this unaddressed, on the theory that since these are just hints and
so don't matter, then the relevant question is: "If this stuff doesn't matter,
why are we spending our time defining this?" 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Tuesday, July 03, 2012 2:54 PM
To: Chuck Lever
Cc: <[email-masked]>
Subject: Re: [nfsv4] Scribe 6/28

It doesn't matter.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 5 Jul 2012 12:29:19 -0400

> provided another hint, or the server detected a better algorithm

Me thinks you aren't terribly serious about that.

First, if the server is just going to diddle with the hint any way it feels like, why bother returning a bitmask that says whether the hint is supported?

Second, there has to be a hard stopping point, after which a hint is no longer effective.  If not, then the server is allowed to hold onto the hint until it reboots or the file is deleted, or even forever.  Since we always want the hint to be "NORMAL" when a file is first opened,  the hint lifetime is limited, by definition, to "until a file is closed, a subsequent hint on the same byte range is issued, or the server reboots, whichever occurs first."  That's our upper bound, so to speak, and we can shorten the lifetime further if that makes sense.

Third, what if an application opens a file and leaves it open?  The server is allowed to forget a hint at any time, you say, which makes it useless for any long-running application.  And, as Dave points out, if servers are allowed to drop a hint during a migration, and the client isn't required to keep sending hints, a hint vanishes when a file system is migrated.

As far as the draft is concerned, I think it is impractical to leave hint lifetime totally up to server implementors.  We do have to address server reboot and migration, at least; and our definition of hints (when a file is opened, hint is "NORMAL") has additional implications.

> Might cause thrashing on the server...

That is a risk of any caching scheme.

---

From: Dean <[email-masked]>
Date: Thu, 05 Jul 2012 16:59:59 -0700

I'll add a few comments to the discussion,

1) In general, the return value is designed to be informational-only, 
but could provide some information as to the current state of the server 
back to the client.  For example, if client specifies 'sequential', and 
the client gets back 'random' or 'normal', then the client knows that 
either the server is busy serving other clients (or simply doesn't 
support 'sequential').  I believe its real usefulness though lies when 
you have a client and server that are tested as a pair with agreed upon 
semantics, and allow the client to depend on the returned hint.  This 
would allow the client to possibly change its behavior based upon the 
return value.  In general though, I think the returned hint has limited 
value for generic clients and servers.

2) Some hints must last at least the lifetime of a stateid, or until 
close, since compound-only information is very limited in its usefulness 
for hints like NOREUSE.  A server can optimize future requests a lot 
more when it knows about access to a file in advance.  For example, if 
data is to be prefetched, should it evict other data to do so or try its 
best to avoid evicting cached data?

3) I agree the spec should have some discussion of the lifetime of the 
hints, but I don't have a hard preference on whether this is written 
into the spec or is merely advice to implementers.

Dean

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 8 Jul 2012 00:29:15 +0000

On Jul 5, 2012, at 6:59 PM, Dean wrote:

> 1) In general, the return value is designed to be informational-only, but could provide some information as to the current state of the server back to the client.  For example, if client specifies 'sequential', and the client gets back 'random' or 'normal', then the client knows that either the server is busy serving other clients (or simply doesn't support 'sequential').  I believe its real usefulness though lies when you have a client and server that are tested as a pair with agreed upon semantics, and allow the client to depend on the returned hint.  This would allow the client to possibly change its behavior based upon the return value.  In general though, I think the returned hint has limited value for generic clients and servers.

I think the return value is a long term investment. It may not be used much at first, but as you state, as clients and servers get better at this, the client can make better decisions.

> 
> 2) Some hints must last at least the lifetime of a stateid, or until close, since compound-only information is very limited in its usefulness for hints like NOREUSE.  A server can optimize future requests a lot more when it knows about access to a file in advance.  For example, if data is to be prefetched, should it evict other data to do so or try its best to avoid evicting cached data?

"must" -> "SHOULD" , but in any case, the server may throw it away.

I also think we can't claim that the server will remember a hint in the NFS layer. I.e., the Linux server could just reuse the API into posix_fadvise(). That would be easy to do and would allow multiple outstanding hints, including NOREUSE. Why would the Linux server implementors want to rewrite posix_fadvise()?

If we want that the NFS layer MUST remember all the applied hints, then I think we need to make this more explicit - which would include an explicit means to release a hint. I.e., think about locks and LOCKU.

Back to migration, I think we can only make this the server's responsibility iff it does remember all hints. But even then, the new server may or may not support the same set of hints. A smart client would still reissue the hints once it detected a migration. And we might need a way to specify in the fs locations info that the servers can support the same set of hints.

> 
> 3) I agree the spec should have some discussion of the lifetime of the hints, but I don't have a hard preference on whether this is written into the spec or is merely advice to implementers.

I think we need to get closer to the semantics of posix_fadvise() for the hints taken from it.

1) All hints are byte ranges - not entire file based. If you want the entire file, then specify it in the call. 

2) Place no limits on the number of outstanding hints.

3) Place no limit on the lifetime of a hint - it gets applied and the underlying OS handles it. If a client wants to issue it per compound, then fine, it does so.