Subject: [nfsv4] [NFSv4.2] Meeting notes for 05/19/11

From: Thomas Haynes <[email-masked]>
Date: Thu, 19 May 2011 13:59:42 -0500

Attendees:

Tom Haynes (NetApp)
Fred Isaman (NetApp)
Sorin Faibish (EMC)
Peter Staubach (EMC)
Dean Hildebrand (IBM)
Andy Adamson (NetApp)
Dave Noveck (EMC)
Rob Thurlow (Oracle)
Trond Myklebust (NetApp)
Chuck Lever (Oracle)

+ IETF Note Well Agreement

   http://www.ietf.org/NOTEWELL.html

+ enum data_content4

Overview of agenda contents

+ ADB, Holes, and pNFS

Initialized ADBs and holes will share similar issues in how sparse files
are implemented.

1) How data flows

Peter remarked this was similar to how setattr/truncate works. 

AI: Tom to review that and see if it addresses issues or raises new ones.

2) Mechanics

sparse versus dense packing

A DS cannot reply about any storage it does not own.

Tom mentioned he removed HOLE_NOINFO - it appears as an error condition.

Dean replied that it was used for the server to inform the client that the
hole was larger than the data requested. 

Tom asked about a sparse map and Dean was against this because of
the change attribute.

The discussion then focused over why a DS can only respond about the
data it owns. In the Interim meeting, it was pointed out that hole information
is metadata and the DS cannot do that.

Tom asked why can't the DS reply about holes in the slice of the file it owns.
(I'll send out a more detailed email later.)

+ Arm not supported

We agreed to add a new error code to help differentiate this case.

+ Heterogeneous SSC and filenames with AUTH_SYS

We could not see the problem here. If the security flavor is AUTH_SYS,
then the implication is that the server trusts everything presented to it.
If the server does not want to trust, then it must use RPCSEC_GSSv3.

AI: Tom to follow up with David Black.

+ BAT

EMC and NetApp will have SSC. 

Oracle and Linux will have FedFS.

Linux and EMC will have IPv6 for pNFS.

+ Bonus item: INITIALIZE

Peter and Trond pointed out that this problem already exists and is handled.

Peter's example was what happens if the first write to a file is at 2G? The server
has to fill in the first 2G with zeros.

Trond's example was a setattr to extend the file.

+ fadvise

Dean brought up a model where we implement the posix standard as the core
and then extend it with the Enterprise Apps requirements.

Tom brought up Spencer's point from the WG alias:

For proposals that overlap, I would ask that the authors/interested parties
work between themselves in an attempt to resolve into a single proposal
or approach.  At a minimum, identify for the rest of us, the main issues
and tradeoffs, etc. 

Dean replied that he thought that had been done on the WG alias already.
He would prefer that we converse about it instead of redoing it on the WG alias.

AI: Tom to make sure fadvise is on the agenda for the next meeting
AI: Dean to invite the author's of the Enterprise App I-D to be present.

---

From: Chuck Lever <[email-masked]>
Date: Thu, 19 May 2011 15:10:34 -0400

> Oracle and Linux will have FedFS.

To be clear, I was asking to see if there are others who might have a FedFS implementation that will be at the Bake-a-thon.  There doesn't seem to be anyone else coming, thus we are not bringing our FedFS implementations.  We already can do testing between the Solaris and Linux implementations internally.

If anyone else has one they'd like to test with us, we will reconsider (at least for the Linux implementation, since I already live in Ann Arbor it would be easy to stop by).

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Dean <[email-masked]>
Date: Fri, 20 May 2011 14:47:47 -0700

> AI: Dean to invite the author's of the Enterprise App I-D to be present.
I just found out that I may be on a plane at the next meeting (June 
2nd), but let's prepare for this discussion anyways.  Once we have the 
meeting (either the next one or the following one), it would be vital 
for the authors of both drafts to attend.

The 2 drafts that are proposing having clients send 'access' hints to 
the server are:

1) Support for posix_fadvise
http://tools.ietf.org/html/draft-hildebrand-nfsv4-fadvise-01
which introduces a new operation FADVISE to support the POSIX fadvise 
and madvise operations

and

2) Extending NFS to Support Enterprise Applications (
(http://tools.ietf.org/html/draft-eisler-nfsv4-enterprise-apps-01)
which introduces 3 new operations (IO_ADVISE, READ_WITH_ADVICE, 
WRITE_WITH_ADVICE) that support a series of new flags that can be sent 
to the server.

My hope is that in NFSv4.2 we can introduce a new operation (called 
IO_ADVISE or FADVISE) that supports:
i) The POSIX standard FADVISE flags as stated by the first draft above
ii) Any additional flags from the enterprise draft that are not already 
covered via the FADVISE flags. (that everyone agrees upon)

The only discussion I know of so far on this list w.r.t. these two 
proposals is found here:
http://www.ietf.org/mail-archive/web/nfsv4/current/msg09172.html

I believe this email discussion raised 2 points that need to be agreed upon:

Point 1: Extra flags
============

 From this discussion, it seems that there are 2 additional flags in the enterprise draft.  Here they are and a summary of the discussion.

1)
    -  IO_ADVISE4_PREFETCH_OPPORTUNISTIC: Stateid holder expects to
       access the data soon; prefetch if it can be done at a marginal
       cost.

Summary statement from Trond: Use case for this? If the client knows that it will want to read the data soon, then when would it not want the server to prefetch the data at any cost?

2)
    - IO_ADVISE4_RECENTLY_USED: The client has recently accessed the
       byte range in its own cache.  This informs the server that the
       data in the byte range remains important to the client.  When the
       server reaches resource exhaustion, knowing which data is more
       important allows the server to make better choices about which
       data to, for example purge from a cache, or move to secondary
       storage.  It also informs the server which delegations are more
       important, since if delegations are working correctly, once
       delegated to a client, a server might never receive another I/O
       request for the file.

Trond: Again, what is the use case? Most clients already cache data that they
know is important using an LRU model. Why would they want the server to
cache the same data?

Dean: This gets into the 2 cache problem, which has been studied with 
somedepth, most notably by Peter H, Dan. M., and local ibmer T. Wong. 
I'dbe very hesitant to include any features describing multi-level 
cachingwithout a lot more thought and effort derived from these people's 
work.Basically, I think there would be a suite of hints that the client 
wouldwant to share with the server regarding multi-level caching, 
andincluding just 1 speculative item is only going to cause problems.

Point 2: New read/write operations
=================================
In addition, there is the issue of introducing new read and write 
operations to atomically include these hints (READ_WITH_ADVICE, 
WRITE_WITH_ADVICE).

In the email thread, summary comment from Trond:
In the case of READ, we are mainly talking about prefetching.

In the case of WRITE, we are talking about whether the server should
write through its read cache or whether it should use an O_DIRECT-like
mechanism in order to do an uncached write.

In both cases, we're talking about hints that constitute a client's best
estimate for how it will be using the data in the future. While that
estimate may indeed change, it is only useful to the server if it is
stable for a non-zero period of time. So how does atomicity with
READ/WRITE help in any way?

Dean