Subject: [nfsv4] New Version Notification for draft-hildebrand-nfsv4-fadvise-01

From: Dean <[email-masked]>
Date: Tue, 29 Mar 2011 11:26:08 -0700

A new version of I-D, draft-hildebrand-nfsv4-fadvise-01.txt has been 
successfully submitted by Dean Hildebrand and posted to the IETF repository.

Filename:          draft-hildebrand-nfsv4-fadvise
Revision:          01
Title:             Support for posix_fadvise
Creation_date:     2011-03-29
WG ID:             Independent Submission
Number_of_pages:   9

Abstract:
This document proposes a new FADVISE operation for NFSv4.2 to support
the posix_fadvise function.  FADVISE will communicate expected
application behavior to the server, allowing servers to optimize
future I/O requests for a file.  The posix_fadvise function is
supported in Linux and many other operating systems and is starting
be widely used by applications. In addition, the FADVISE operation
can communicate other application directions such as the use of
direct I/O.
-------------

As promised, here is an ID that allows NFS to support the POSIX fadvise 
operation as defined here:
http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_fadvise.html

This ID also allows support for sending I/O access hints when a client 
uses direct I/O or at any other point when a client feels it could be 
useful.

This sets out the minimum number of client initiated *hints* to the 
server that we must support.  These hints can be extended as needed for 
other requirements.

Hope everyone had fun in Prague,

Dean

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Tue, 29 Mar 2011 20:43:35 -0700

The possible hints have direct mapping to POSIX FADVICE. Mike and I were
looking at integrating the FADVISE into his enterprise apps draft
proposal for IO_ADVISE
(http://tools.ietf.org/html/draft-eisler-nfsv4-enterprise-apps-01).

Summary of what we thought:

   o  FADVISE_NORMAL - Specifies that the application has no advice to
      give on its behavior with respect to the specified data. It is
      the default characteristic if no advice is given for an open
      file.

[Pranoop]: Do we really need this? If you'd like server to not know any
other type of access, there really isn't a need to convey anything.
Saves extra processing of operations by the servers.

   o  FADVISE_SEQUENTIAL - Specifies that the application expects to
      access the specified data sequentially from lower offsets to
      higher offsets.

[Pranoop]: Needs to be merged with the IO_ADVISE arguments for
SEQUENTIAL:

         - IO_ADVISE4_SEQUENTIAL_CACHE: Sequential access to data
expected.
         The server should leave data in its cache.
         - IO_ADVISE4_SEQUENTIAL_DONTCACHE: Sequential access to data
         expected.  The server does not need to leave data in its cache.

Next,

   o  FADVISE_RANDOM - Specifies that the application expects to access
      the specified data in a random order.

[Pranoop]: Same as IO_ADVISE argument    IO_ADVISE4_RANDOM.

         - IO_ADVISE4_RANDOM: Random access to data expected.   

[Pranoop]: These are good to have.

   o  FADVISE_WILLNEED - Specifies that the application expects to
      access the specified data in the near future.

However, there is an overlap between FADVISE_DONTNEED and the
IO_ADVISE4_INTENT_TO_WRITE.

   o  FADVISE_DONTNEED - Specifies that the application expects that it
      will not access the specified data in the near future.

That said, do we see a value for IO_ADVISE_INTENT_TO_READ. This could
potentially pass hints about the type of access as well.

[Pranoop]: Mike's proposal does not have this. So, it would be good to
keep it.

   o  FADVISE_NOREUSE - Specifies that the application expects to access
      the specified data once and then not reuse it thereafter.

[Pranoop]: We really don't see the use case for this. This to me sounds
like can be simulated by FADVISE_DONTNEED.

[Pranoop]: Three additional ones that we need to bring in from Mike's
document are:

   -  IO_ADVISE4_PREFETCH: Stateid holder expects to access the data
      soon; prefetch data in preparation.

   -  IO_ADVISE4_PREFETCH_OPPORTUNISTIC: Stateid holder expects to
      access the data soon; prefetch if it can be done at a marginal
      cost.

   - IO_ADVISE4_RECENTLY_USED: The client has recently accessed the
      byte range in its own cache.  This informs the server that the
      data in the byte range remains important to the client.  When the
      server reaches resource exhaustion, knowing which data is more
      important allows the server to make better choices about which
      data to, for example purge from a cache, or move to secondary
      storage.  It also informs the server which delegations are more
      important, since if delegations are working correctly, once
      delegated to a client, a server might never receive another I/O
      request for the file.

Further,

[Pranoop]: This document doesn't talk about passing the hints atomically
with the READ/WRITE operations. Just curious, if you don't see any value
with such changes to READ/WRITE. Mike's proposal proposes those as well.

- Pranoop

> -----Original Message-----
> From: Dean [mailto:[email-masked]]
> Sent: Tuesday, March 29, 2011 8:26 PM
> To: [email-masked]
> Subject: [nfsv4] New Version Notification fordraft-hildebrand-nfsv4-
> fadvise-01
> 
http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_fadvise.
> html
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 30 Mar 2011 10:53:05 +0200

> Saves extra processing of operations by the servers.

Applications sometimes want to switch between several different access
modes. You need the above in order to switch back to default behaviour
after you are done using the FADVISE_SEQUENTIAL/FADVISE_RANDOM modes.

> like can be simulated by FADVISE_DONTNEED.

Nope. In the READ case, this means 'please prefetch this into your cache
now, but I'm only going to ask you for it once'.
I think this is the mode that Dean also wants to use to signify that the
server might want to use direct I/O (uncached) writes.

>       soon; prefetch data in preparation.

This sounds exactly like the definition of the posix
FADVISE_WILLNEEED...

>       access the data soon; prefetch if it can be done at a marginal

Use case for this? If the client knows that it will want to read the
data soon, then when would it not want the server to prefetch the data
at any cost?

>       delegated to a client, a server might never receive another I/O

Again, what is the use case? Most clients already cache data that they
know is important using an LRU model. Why would they want the server to
cache the same data?

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 30 Mar 2011 11:36:23 +0200

> with such changes to READ/WRITE. Mike's proposal proposes those as well.

This has me really confused.

In the case of READ, we are mainly talking about prefetching.

In the case of WRITE, we are talking about whether the server should
write through its read cache or whether it should use an O_DIRECT-like
mechanism in order to do an uncached write.

In both cases, we're talking about hints that constitute a client's best
estimate for how it will be using the data in the future. While that
estimate may indeed change, it is only useful to the server if it is
stable for a non-zero period of time. So how does atomicity with
READ/WRITE help in any way?

NetApp

---

From: Dean <[email-masked]>
Date: Thu, 31 Mar 2011 12:13:42 -0700

>> (http://tools.ietf.org/html/draft-eisler-nfsv4-enterprise-apps-01).
Yes, I commented a few weeks ago on the lack of description of FADVISE 
in this draft, and wondered if we should merge this FADVISE draft with 
the useful I/O hints in that draft.

One general comment is that these hints are about recommendations for 
the server, not orders for the server.  The server will always have the 
last say on what it will do and not do on future i/o requests.
> after you are done using the FADVISE_SEQUENTIAL/FADVISE_RANDOM modes.
Yes, exactly.
>>           expected.  The server does not need to leave data in its cache.
Aren't these IO_ADVISE hints just a combination of fadvise sequential 
and willneed/dontneed?  So they don't seem necessary.
>> IO_ADVISE4_INTENT_TO_WRITE.
One thing I notice in posix fadvise is that they don't convey write or 
read.  I assume that this is because this type of hint is already 
indicated in the OPEN of the file.  What is the usecase for going beyond 
this?

>> potentially pass hints about the type of access as well.
Same as above regarding _TO_READ.
> server might want to use direct I/O (uncached) writes.
Yes.  Let's also remember that we don't want to only support 83% of a 
posix operation.

> data soon, then when would it not want the server to prefetch the data
Agreed.
> know is important using an LRU model. Why would they want the server to
This gets into the 2 cache problem, which has been studied with some 
depth, most notably by Peter H, Dan. M., and local ibmer T. Wong.  I'd 
be very hesitant to include any features describing multi-level caching 
without a lot more thought and effort derived from these people's work.  
Basically, I think there would be a suite of hints that the client would 
want to share with the server regarding multi-level caching, and 
including just 1 speculative item is only going to cause problems.  This 
means that I don't believe it is ready for the train...

Dean

---

From: Dean <[email-masked]>
Date: Thu, 31 Mar 2011 12:16:16 -0700

> READ/WRITE help in any way?
In general I agree with Trond, I don't see the use case.  One item I 
could see being useful to pass with READ operations was proposed by Sam 
F, which is a flag to indicate whether the READ is optional or not.  
IOW, is the read request a prefetch/readahead and therefore the server 
can ignore the request, return an error, and not cause any errors to the 
application.  This is a rather speculative optimization though...
Dean
> -- Trond Myklebust

---

From: Dean <[email-masked]>
Date: Thu, 31 Mar 2011 14:06:51 -0700

Copying list as I assume you meant to.

On 3/31/11 1:11 PM, Nico Williams wrote:
> On Thu, Mar 31, 2011 at 2:16 PM, Dean<[email-masked]>  wrote:
>> In general I agree with Trond, I don't see the use case.  One item I could
>> see being useful to pass with READ operations was proposed by Sam F, which
>> is a flag to indicate whether the READ is optional or not.  IOW, is the read
>> request a prefetch/readahead and therefore the server can ignore the
>> request, return an error, and not cause any errors to the application.  This
>> is a rather speculative optimization though...
> You might be writing and the server might want to aggregate writes (as
> many filesystems nowadays do).  But the client might know something
> useful about how best to layout the file on disk, that is, whether it
> will pay for the server to try to keep as much of the file's data
> contiguous as possible.  Writes might be random, but reads sequential,
> and even if the client is only writing (the app opened the fildes with
> O_WRONLY, say) the app could still advise the server about read
> patterns.
>
> This could be useful for database applications that write in random
> patterns and read in random patterns but often enough also perform
> sequential reads.
Ok, so you are saying there will be different access patterns for read 
and write.  This isn't a use case for per I/O request flags, but for per 
I/O-type flags.  As Trond says, the usefulness comes from the pattern 
being stable for a non-zero period of time.  So, in your example, it 
might be possible for the db to simply change its FADVISE from random to 
sequential and back again as needed.  Are DB clients really doing 
sequential reads interspersed with random writes over the exact same 
byte range simultaneously?

I still don't see the use case for having per-I/O-type hints for the 
same byte-range.

Dean

> Nico

---

From: Nico Williams <[email-masked]>
Date: Thu, 31 Mar 2011 16:39:59 -0500

> Copying list as I assume you meant to.

Ah, yes, I did.

>> O_WRONLY, say) the app could still advise the server about read

I would add that a server that wants to aggregate writes might still
do that but schedule the file for aggressive defragmentation given the
client's advice.

Also, this might be better signaled via a file attribute than via an
operation, or at least it'd benefit apps that don't call fadvise() to
have such an attribute.

> Ok, so you are saying there will be different access patterns for read and
> write.  This isn't a use case for per I/O request flags, but for per
> I/O-type flags.  As Trond says, the usefulness comes from the pattern being
> stable for a non-zero period of time.  So, in your example, it might be
> possible for the db to simply change its FADVISE from random to sequential
> and back again as needed.  Are DB clients really doing sequential reads
> interspersed with random writes over the exact same byte range
> simultaneously?

Simultaneously?  No, they're bound to lock around their I/Os, but
given byte range locking around their I/Os, yes, definitely.  Consider
something like SQLite3, that places many tables in one file: some
processes/threads might be doing queries that involve full table scans
while others do updates.  SQLite3 has relatively low write
concurrency, so the sequential read pattern following a random write
pattern will not be concurrent with each other, but may come soon
after the write, whereas a DB that supports row-level locking (think
PostgreSQL) will probably feature much more random write and
sequential read operations.

And, of course, a DB like SQLite3 might sometimes use sequential read
patterns and other times random read patterns -- it all depends on how
a SQL statement is handled by the RDBMS' optimizer (it can be faster
to scan a table even when you have suitable indexes, in which case you
get sequential reads, whereas you'd get random reads if the optimizer
uses indexes instead -- the choice of query plan will vary by query,
and the application might do many different queries randomly).

> I still don't see the use case for having per-I/O-type hints for the same
> byte-range.

Yeah, probably not.  I mean, a SQL DB can have a bunch of metadata
that doesn't require fast sequential reads (not least because such
metadata might be small enough to cache easily), interspersed with a
lot more data that does require fast sequential reads (e.g., b-tree
leaf nodes being data and interior nodes being metadata).  But one
might as well say all of it needs fast sequential reads if the
metadata is a small portion of the whole file.

Nico
--