Subject: [nfsv4] A more coherent data cache for NFS4.3 clients

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 10:33:07 -0700

The objective of this feature is to improve NFS client caching for NFS4.3
I would like to find out from the group if one, others see this as a 
useful 
addition to NFS, and two, if extending the delegations feature is the 
right 
approach. 

The requirements as I see them is to make NFS client caching possible 
while
maintaining its coherency without changing the NFS client applications. 
Have 
better synchronization of data among applications on different NFS clients 

while caching data.

The options that we have today for more synchronization of data among NFS 
clients is direct IO which means no data caching. With delegations we can 
have 
caching on multiple NFS client readers or a single NFS client writer. 
 
It would be nice to make better use of delegations and more specifically 
the 
combination of read write delegation. In general the just read delegations 
are 
useful but the combination with write delegation are less useful for most 
cases.

Selecting delegation as the synchronization method and not byte range 
locks
have few reasons. The first objective is to not block applications from 
read or
write but to just synchronize the updates, this extension need to coexist 
with
current delegation and should be considered as a more refined subset of 
the 
current  delegations, and finally because we don't want to requiter any 
changes 
to the application. 

One problem with delegation is that it is server and not client initiated, 
so we 
might need to add some mount options or hints for the applications to 
request 
this new behavior, or adding the option for the NFS client to request a 
byte range 
delegation.

Delegation will be extended to include a byte range and delegation 
callback
will have to specify the part of the file that were changed by a writer or 
being 
requested by  a reader.

It might be required that the byte range be on block boundary. The byte 
range
might start with full file delegations, but it might be reduced to the 
range that
represent the cached portion of the file. We will have to put some limit 
to the
number of byte ranges to keep resources usage under control.

The applications that can benefit from this feature are HPC application, 
multi-node 
database programs, web server cluster, etc.

case 1:
Lets consider the case of multiple readers and one writer. In this 
multi-reader-one-writer
mode all clients will keep both the delegations and the cache and will 
start with a full 
file byte range.
When a reader makes a read of some byte range from the server the server 
will make a 
callback to the writer, if it is holding a delegation for the requested 
read byte range, 
requesting that this range be flushed to the server, the writer will flush 
the changed 
byte range if the range is dirty before the server can respond to the 
reader.
The writer might also consider give up that byte range to avoid more call 
backs.

case 2:
Again all client in this multi-reader-one-writer mode will maintain their 
caches.
When the writer writes to the server all readers will be notified of the 
changed byte range.
At this point the reader can purge that range from its cache if it was 
there in the first place
or read that range from the server to keep its cache up to date. It can 
also give up the 
delegation if it is not interest in caching this range.

case 3:
There are HPC application that split to file to segments that each node is 
working on.
It will be nice to give each client a write delegation for the portion of 
the file that is will
update, a good hint would be when the client punches a hole.

Marc.

---

From: Rick Macklem <[email-masked]>
Date: Mon, 20 May 2013 14:22:11 -0400 (EDT)

Marc Eshel wrote:
> maintaining its coherency without changing the NFS client
> applications. Have
> clients is direct IO which means no data caching. With delegations we
> can have
> It would be nice to make better use of delegations and more
> specifically the
> combination of read write delegation. In general the just read
> delegations are
> useful but the combination with write delegation are less useful for
> most cases.
> 
> have few reasons. The first objective is to not block applications
> from read or
> write but to just synchronize the updates, this extension need to
> coexist with
> current delegation and should be considered as a more refined subset
> of the
> One problem with delegation is that it is server and not client
> initiated, so we
> this new behavior, or adding the option for the NFS client to request
> a byte range
> delegation.
> 
> will have to specify the part of the file that were changed by a
> writer or being
> requested by a reader.
> 
> It might be required that the byte range be on block boundary. The
> byte range
> represent the cached portion of the file. We will have to put some
> limit to the
> The applications that can benefit from this feature are HPC
> application, multi-node
> When a reader makes a read of some byte range from the server the
> server will make a
> callback to the writer, if it is holding a delegation for the
> requested read byte range,
> requesting that this range be flushed to the server, the writer will
> flush the changed
> The writer might also consider give up that byte range to avoid more
> call backs.
> 
So, once the reading client reads a byte range from the server, the
write client can no longer modify that byte range? (Sounds like a
downgrade to a read delegation to me?)

If that isn't the case...
What if the client with the "read delegation" has data cached. Why
would it "make a read of some byte range from the server..."? It seems
that, if the write client is allowed to continue to modify the byte range,
the read client(s) will need to check with the server and cause a callback
to the write client upon every read. (Sounds a lot like direct I/O to me.)

> case 2:
> Again all client in this multi-reader-one-writer mode will maintain
> their caches.
> When the writer writes to the server all readers will be notified of
> the changed byte range.
> or read that range from the server to keep its cache up to date. It
> can also give up the
Which implies that the writer will have to do all writes directly to
the server (or flush them to the server before other clients will see
the changes, if you prefer.

> case 3:
> There are HPC application that split to file to segments that each
> node is working on.
> It will be nice to give each client a write delegation for the portion
> of the file that is will
I can see a use for having a "I can cache this byte range of the file"
token. I use the term "token", since that was what DFS (the massive child
of AFS that never quite worked) called it. In DFS, there was a "token" held
by a client, which said that client could cache file data. (I can't recall
if there was a read-only/shared version of the token.) They found that for
read-write sharing, the token would pingpong between clients and they ended
up making the server hold the token for that case-->direct I/O.
I can see a byte range token being useful for cases where the clients are
reding/writing non-overlapping byte ranges. For the same byte range, my
concern is that all the flushing and cache invalidation will end up somewhere
near direct I/O. (I'd suggest you look for some real apps. that run on
multiple clients with one being a writer and then see what mechanism works
for those.)

Since delegations allow opens and byte range locks to be done locally on
a client, I think that extending delegations this way would be too complex
and confusing. (Implementing delegations is confusing enough already.;-)
That was why I'd suggest a separate entity for caching, if you proceed
with this.

rick

> Marc.
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Myklebust, Trond" <[email-masked]>
Date: Mon, 20 May 2013 18:36:50 +0000

On Mon, 2013-05-20 at 10:33 -0700, Marc Eshel wrote:
> current  delegations, and finally because we don't want to requiter
> any changes 
> to the application. 
> 

Aside from the cases where O_DIRECT and locking are active, when would
the client not want to request a delegation for something that it is
reading?

> update, a good hint would be when the client punches a hole. 

???? Why is that a hint?

Either way, please see

https://datatracker.ietf.org/doc/draft-myklebust-nfsv4-byte-range-delegations/

for a draft implementation of the subfile delegation concept.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 20 May 2013 19:52:50 +0000

While it is good to consider areas for future work, I'd like to avoid a discussion about
what NFSv4.3 looks like until we deliver both a NFSv4.2 spec and some interoperating
implementations.

I feel we've gotten away from tracking them both at the same time.

With some of the work going on in the Linux camp, I think we are finally starting
up that effort for NFSv4.2.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 13:04:40 -0700

I was not aware of your draft I will look it over and reconsider my 
proposal.
Thanks, Marc.

"Myklebust, Trond" <[email-masked]> wrote on 05/20/2013 
11:36:50 AM:

> From: "Myklebust, Trond" <[email-masked]>
> To: Marc Eshel/Almaden/IBM@IBMUS, 
> Cc: "[email-masked]" <[email-masked]>
> Date: 05/20/2013 11:37 AM
> Subject: Re: [nfsv4] A more coherent data cache for NFS4.3 clients
>

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 13:12:46 -0700

Rick Macklem <[email-masked]> wrote on 05/20/2013 11:22:11 AM:

> From: Rick Macklem <[email-masked]>
> Cc: [email-masked]
> Date: 05/20/2013 11:22 AM

It can the question is can it continue to cache that byte range and that 
depends on if it decides to keep the delegation for that range or not.
 
> to the write client upon every read. (Sounds a lot like direct I/O to 
me.)

Maybe the case is not described in enough details but the point is that if 
a client hold read delegation for some range and that range is changed the 
reader will get notified, and if a a writers hold a delegation on a range 
that has readers it can cache but it must be a write through cache.
 
> the changes, if you prefer.

Yes, only in this new combined mode.

> multiple clients with one being a writer and then see what mechanism 
works

My idea was to use Token like distributed file system and cluster fs like 
GPFS use, but I was trying to see it will feet with delegations.
 
> That was why I'd suggest a separate entity for caching, if you proceed

We will have to deal with both delegation and whatever mechanism we come 
up with so I am not sure what is more complicated to separate features or 
one combined one, it will be complicated either way.  I will look at Trond 
draft and reconsider how to proceed. I think we should talk some more 
about requirement before choosing implementation.
Thanks, Marc.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 13:15:21 -0700

I think our process is slow enough with adding serialization to it.
Marc. 

[email-masked] wrote on 05/20/2013 12:52:50 PM:

> From: "Haynes, Tom" <[email-masked]>
> To: nfsv4 list <[email-masked]>, 
> Date: 05/20/2013 12:53 PM
> Subject: [nfsv4] NFSv4.3 [was Re: A more coherent data cache for 
> NFS4.3 clients]
> Sent by: [email-masked]
>

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 13:21:47 -0700

I think our process is slow enough without adding serialization to it. 
Marc. 

While it is good to consider areas for future work, I'd like to avoid a 
discussion about
what NFSv4.3 looks like until we deliver both a NFSv4.2 spec and some 
interoperating
implementations.

With some of the work going on in the Linux camp, I think we are finally 
starting
up that effort for NFSv4.2.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Nico Williams <[email-masked]>
Date: Mon, 20 May 2013 15:33:13 -0500

I don't think you can get POSIX write visibility rules without
resorting to Lustre-style byte-range locking (in a namespace that is
distinct from the normal byte range locks).

There are applications that expect better than NFS semantics.  And NFS
has tripped up a few applications (just look at Maildir and things of
that sort).  In particular there are several POSIX behaviors that apps
depend on:

 - O_EXCL | O_CREAT (fixed in NFSv3, IIRC)
 - rename atomicity (fixed in NFSv4?)
 - write visibility (that a write() to a part of some file will be
visible to any read()s that can be proven to occur after the write())
 - O_APPEND aotmicity (although, to be fair, POSIX says very little
about just what can be expected here)

Possibly also:

 - mkdir() atomicity
 - rmdir() atomicity
 - visibility of length-extending write()s via stat() (this is a royal
PITA for something like pNFS or Lustre)

Write visibility is harder to depend on, but apps that use any kind of
locking tend to (since use of locking is one way to prove that a read
occurs later than a write).

Nico
--

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 13:54:31 -0700

Can you please point me to a description of the Luster-style byte-range 
locking.

I was thinking mainly about performance improvement that can be achieved 
by better caching given that delegation is based on optional features and 
behavior. If we want to fix the POSIX write correctness issue we will have 
to mandate specific behavior that I did not consider but can be added to 
the objectives.

Thanks, Marc.

From:   Nico Williams <[email-masked]>
To:     Marc Eshel/Almaden/IBM@IBMUS, 
Cc:     [email-masked]
Date:   05/20/2013 01:36 PM
Subject:        Re: [nfsv4] A more coherent data cache for NFS4.3 clients

Possibly also:

 - mkdir() atomicity
 - rmdir() atomicity

Nico
--

---

From: Rick Macklem <[email-masked]>
Date: Mon, 20 May 2013 17:28:07 -0400 (EDT)

> occurs later than a write).
It is my understanding that NFSv4 clients are required to flush writes
when an exclusive lock is unlocked and check with the server when a
lock is acquired to check if cached data is still valid.

As such, I think "write visibility" is fixed, if it is tied to byte
range locking. Without byte range locking, the FreeBSD client will
maintain this "approximately, based on attribute cache timeout", if
the writing client app. does an fsync(). The FreeBSD collective is
debating relaxing this, since it does result in a big performance
hit for things like the loader, which writes in non-block sized
non-contiguous chunks. I think that tying this to byte range
locking may be appropriate.

rick

---

From: Nico Williams <[email-masked]>
Date: Mon, 20 May 2013 16:07:30 -0500

> Can you please point me to a description of the Luster-style byte-range

Lustre clients hold locks for the byte ranges they read or write to.
When a client wants to write it has to first acquire a lock to the
range it wants to write to, and this may require recalling locks from
other clients.  These locks are held as long as the client holds the
file open (which may be longer than the application holds the file
open).

This can get slow when two or more clients play ping-pong over the
same byte range, in which case performance goes down, of course.

> I was thinking mainly about performance improvement that can be achieved by
> better caching given that delegation is based on optional features and
> to mandate specific behavior that I did not consider but can be added to the
> objectives.

Right, clients ought to be able to keep caches for as long as they
retain the corresponding byte range locks.  If a client wants to write
then the holders of the locks have to be called back to return the
lock(s) (or punch holes in them) so the write can take place.

This is probably not what you want.

Another possibility would be to have a file content cache
synchronization protocol where clients cache file content as long as
they want but must synchronize it at certain times (e.g., when the app
acquires a byte range lock, or if the client gets a notification from
a server, say).  This is not exactly trivial (especially if the
synchronization requires lots of per-client server-side state), but it
has the benefit of not having to call back any clients at write time.
And it gets somewhat less than POSIX write visibility, but you can
always approximate via the app triggering synchronization by acquiring
byte range locks.

A stateless synchronization protocol could be based on the rsync
protocol, applying it to the range of interest to the client app.

There are optimizations.  For example, the synchronization can be
deferred to after getting the locks, that way a client could batch
multiple locks, then synchronize.

In any case, the first question is: what write visibility semantics
are you after?

Nico
--

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 21 May 2013 01:04:39 +0000

The process as you call it is "specification and working code".

I'd like to see us not revisit the specification several times because we
do not have implementations working.

We started NFSv4.2 without understanding what was broken in NFSv4.1.
We had a window to fix things.

If we start NFSv4.3 without any working implementations of NFSv4.2,
then are we going to have a chance to feedback into NFSv4.3?

Or in other words, perhaps we wouldn't need microversioning
discussions if we were able to fix issues before new features?

BTW - I think NFSv4.2 would have gone faster if we had more mature
NFSv4.1 implementations. But people were too busy trying to ship
NFSv4.1 and could not prototype NFSv4.2 systems.

On May 20, 2013, at 1:15 PM, Marc Eshel <[email-masked]<mailto:[email-masked]>> wrote:

[email-masked]<mailto:[email-masked]> wrote on 05/20/2013 12:52:50 PM:

> From: "Haynes, Tom" <[email-masked]<mailto:[email-masked]>>
> To: nfsv4 list <[email-masked]<mailto:[email-masked]>>,
> Sent by: [email-masked]<mailto:[email-masked]>
>
> [email-masked]<mailto:[email-masked]>
_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 19:08:23 -0700

I don't see how not talking about future work will help with 4.2, I am 
talking about 4.3 because we can not add any more to 4.2. You can call it 
future work if you are not comfortable calling it 4.3. It is not like you 
have a group of programmers at your command and you will not let them do 
anything new before they finish the old version.The discussion is just 
starting and will be awhile before we have a clear draft to implement. We 
should have a draft and an implementation before it should be considered 
for next version. It looks like there is interests in the topic so I think 
we should be able to continue the discussion. If there are more important 
topics for the coming meeting than I can wait for the next one. 
Marc.

From:   "Haynes, Tom" <[email-masked]>
To:     Marc Eshel/Almaden/IBM@IBMUS, 
Cc:     nfsv4 list <[email-masked]>
Date:   05/20/2013 06:21 PM
Subject:        Re: [nfsv4] NFSv4.3 [was Re: A more coherent data cache 
for NFS4.3      clients]

On May 20, 2013, at 1:15 PM, Marc Eshel <[email-masked]> wrote:

---

From: "Welch, Brent" <[email-masked]>
Date: Tue, 21 May 2013 04:00:09 +0000

While I agree that the combination of byte-range locking and cache flushing is
one way to guarantee write visibility, I will point out that it is an expensive solution.
It requires the  client to do a round trip with the server before every write
(i.e., to obtain the covering byte-range lock) to ensure that it sees current data.

If instead you have a callback-based protocol, the client can cache data until notified
by the server that its cache is out-of-date.  That results in a protocol where the number
of messages is proportional to the amount of sharing by concurrent clients.  No sharing -
no messages.  Lots of fine grain sharing - lots of messages.

For example, consider the loader example.  No application has any business trying to read
(or execute!) from a file that is being assembled by the loader.  One solution is  to take
a byte-range lock over the entire file.  I suppose this argues against my transparent,
callback based scheme, but  I don't think it is the best compelling example.

Compelling examples include "reader chases writer".  If there is a writer dumping a file that
represents, e.g., a log of packets or image data, then there can be a reader consuming that
file.  The reader needs to have a reasonable view of the end of file such that when it reads
up to that point it gets the data.  You really don't want the writer to do an extra round
trip (remember, I'm always thinking pNFS where the writer is going to storage and by-passing
the metadata server).  Instead, you want the writer to occasionally commit the end-of-file
position, and have the reader be able to read up to that point and get valid data.  The reader
could have a "callback" on the file attributes, such that it gets notified whenever the writer
updates those attributes.  The callback, or delegation, causes the metadata server to notify
the  reader about the new eof attribute value.  Otherwise, the client has to poll.  And, I don't
think byte range locks help very much in this example, except to add overhead.

Another example, which is harder, is a shared HDF5 file.  This file format is a "file system
inside a file", which has metadata and 4-byte pointers inside the header.  The writer adds chunks
and then updates the header with a 4-byte write.  A reader could be traversing data structures
inside the metadata to find chunks.  Yuck.  OK, for that you want byte-range locks, although really
I'd rather have some higher level API than "traversing a shared data structure on slow storage
via byte-level reads and writes."

Anyway - to summarize, if you  are interested in more efficient cache consistency protocols, you
want a callback scheme where the message overhead is proportional to the sharing that actually
occurs.  The byte-range locking protocol is instead proportional to the I/O that occurs.

--
Brent

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Rick Macklem
Sent: Monday, May 20, 2013 2:28 PM
To: Nico Williams
Cc: [email-masked]
Subject: Re: [nfsv4] A more coherent data cache for NFS4.3 clients

rick

---

From: Marc Eshel <[email-masked]>
Date: Mon, 20 May 2013 21:15:10 -0700

Nico Williams <[email-masked]> wrote on 05/20/2013 02:07:30 PM:

> Date: 05/20/2013 03:06 PM

Yes, I was hoping to have something more dynamic where you start with a 
full file
byte range and than return the conflicting portions or keep them and get 
notifications
of changes.
 
> always approximate via the app triggering synchronization by acquiring

I was hoping to not depend on app locks or require any changes to app.
 
> In any case, the first question is: what write visibility semantics

It would be nice to get POSIX write behavior but than we will have to come 
up with a mechanism that always works and not best effort like delegation 
is.

---

From: Nico Williams <[email-masked]>
Date: Tue, 21 May 2013 00:13:16 -0500

> (i.e., to obtain the covering byte-range lock) to ensure that it sees current data.

Writers need only contend with each other if you don't care about
using only clocks as a method of proving a read follows/does not
follow a write (IIRC POSIX speaks in terms of proof that a read comes
after a write, but does not elaborate).  Now we're back to depending
on applications doing byte range locking to synchronize.

Pick your poison, then design the protocol for it :)

---

From: Rick Macklem <[email-masked]>
Date: Tue, 21 May 2013 09:53:38 -0400 (EDT)

> (i.e., to obtain the covering byte-range lock) to ensure that it sees
Yes. My assumption (which I didn't state) was that the apps. would already
be doing the byte range locking, since they care about read/write sharing
of some file(s).

> callback based scheme, but I don't think it is the best compelling
Right. It doesn't make sense for the loader to byte range lock the
file, since it won't be read until sometime after the loader is done
and exits. As such "close to open" consistency is the most it will
require. That is why the FreeBSD collective will probably optionally
loosen the current NFS client caching model for performance reasons.
(They have already made "close-to-open" consistency optional, following
 what other NFS clients have done. Just to be clear, "loosen" doesn't
 refer to dropping close-to-open entirely. It refers to the current
 client code, which flushes a write to the server when it sees another
 non-contiguous write for the same file. There was a vague assumption
 that a non-sequential writers might imply read/write sharing.)

> occurs. The byte-range locking protocol is instead proportional to the
Agreed. The big question to me is "if you are interested...", which might
be stated as "should NFS mounts exhibit write visibility all the time or
should apps. be expected to use certain system calls to achieve consistency
when they require it or ???".

Although way back when I did a callback style cache coherency algorithm,
no one cared and it eventually was deleted from the BSDen source trees.
As such, I'll tend to be somewhat skeptical of a need for this.

rick
ps: I also feel that Tom's point was appropriate, since I think that
    implementation experience is getting way behind specification at this point.

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 21 May 2013 10:46:15 -0400 (EDT)

All of this, ++.

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: "Noveck, David" <[email-masked]>
Date: Tue, 28 May 2013 11:57:50 -0400

Sorry to be late responding to this, but I've just been out of internet-contact for a while.

> do not have implementations working.

Agree wholeheartedly.  I'd say that the best way to achieve that is not to
consider the spec finished until we have those working implementations.

I know people are concerned that serialization of that sort can slow thing down, but in the end it results in a more efficient process.

> We started NFSv4.2 without understanding what was broken in NFSv4.1.

If we couldn't have started work on v4.2 until we understood what was broken in v4.1, that would have slowed things down.  There are many features in v4.1 that are not implemented yet.

Similarly, if we had waited to start work on v4.1 until we understood what was broken in v4.0, then we would be ready to start v4.1 in 2H2013.  Some issues:

*         Internationalization

*         Migration

So how can one reconcile the need to serialize with the need make more-than-glacial progress?

As way of stimulating discussion (wherever it occurs), let me offer the following opinions about serialization alternatives.

Good serialization:

*         Not considering minor version x complete until you have implementations.

*         Not considering the list of features/elements for a particular minor version decided until there are fairly complete specifications that potential implementers consider adequate to serve as a basis for potential implementations.

Bad serialization:

*         Not allowing discussion of and work on specification of features for future minor versions until the current version is complete.

*         Not starting serious work on a potential feature until you are sure that it can be done within the putative time-frame for the next minor version.

> discussions if we were able to fix issues before new features?

We then wouldn't need microversioning targeted toward future minor
versions, but we would still need some mechanism to deal with the consequences
of the previous way of doing things.

*         Given that v4.1 exists, you would need some way of fixing v4.0 problems unless you are sure that none exist, and I'm reluctant to assume that none exist.

*         For problems in v4.1, if you don't do  microversioning, you would have to rename the current v4.2 to be v4.3 and reserve v4.2 for a fixes-to-v4.1 minor version.  Not a bad idea from a technical perspective but I suppose Spencer and Beepy might have trouble selling it to the IESG.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Monday, May 20, 2013 9:05 PM
To: Marc Eshel
Cc: nfsv4 list
Subject: Re: [nfsv4] NFSv4.3 [was Re: A more coherent data cache for NFS4.3 clients]

---

From: Marc Eshel <[email-masked]>
Date: Tue, 28 May 2013 11:24:16 -0700

I think that I already responded to this concern but I will try again. It 
was my mistake to call it 4.3. At this point it is just future work driven 
by a personal draft, so the editor of the spec should not be concern about 
extra work for now. When the draft looks usable and there is an 
implementation, only than it will be merged to the current version that 
the group is working on. So why bring it to the list? because I want to 
make sure that I am working on something of interest to the group that 
might get accepted when it is ready, I already got good input, like the 
fact that Trond already started a draft for this work, and that we need 
posix write semantics. Isn't that the process to adding features?
Marc.

From:   "Noveck, David" <[email-masked]>
To:     "Haynes, Tom" <[email-masked]>, Marc 
Eshel/Almaden/IBM@IBMUS, 
Cc:     nfsv4 list <[email-masked]>
Date:   05/28/2013 09:12 AM
Subject:        RE: [nfsv4] NFSv4.3 [was Re: A more coherent data cache 
for     NFS4.3  clients]

Sorry to be late responding to this, but I?ve just been out of 
internet-contact for a while. 
 
> do not have implementations working.
 
Agree wholeheartedly.  I?d say that the best way to achieve that is not to
consider the spec finished until we have those working implementations.
 
I know people are concerned that serialization of that sort can slow thing 
down, but in the end it results in a more efficient process.
 
> We started NFSv4.2 without understanding what was broken in NFSv4.1.
 
If we couldn?t have started work on v4.2 until we understood what was 
broken in v4.1, that would have slowed things down.  There are many 
features in v4.1 that are not implemented yet.
 
Similarly, if we had waited to start work on v4.1 until we understood what 
was broken in v4.0, then we would be ready to start v4.1 in 2H2013.  Some 
issues:
·         Internationalization
·         Migration
 
So how can one reconcile the need to serialize with the need make 
more-than-glacial progress?
 
As way of stimulating discussion (wherever it occurs), let me offer the 
following opinions about serialization alternatives.
 
Good serialization:
·         Not considering minor version x complete until you have 
implementations.
·         Not considering the list of features/elements for a particular 
minor version decided until there are fairly complete specifications that 
potential implementers consider adequate to serve as a basis for potential 
implementations.

Bad serialization:
·         Not allowing discussion of and work on specification of features 
for future minor versions until the current version is complete.
·         Not starting serious work on a potential feature until you are 
sure that it can be done within the putative time-frame for the next minor 
version.
 
> discussions if we were able to fix issues before new features?
 
We then wouldn?t need microversioning targeted toward future minor
versions, but we would still need some mechanism to deal with the 
consequences
of the previous way of doing things. 
·         Given that v4.1 exists, you would need some way of fixing v4.0 
problems unless you are sure that none exist, and I?m reluctant to assume 
that none exist.
·         For problems in v4.1, if you don?t do  microversioning, you 
would have to rename the current v4.2 to be v4.3 and reserve v4.2 for a 
fixes-to-v4.1 minor version.  Not a bad idea from a technical perspective 
but I suppose Spencer and Beepy might have trouble selling it to the IESG.

 
From: [email-masked] [mailto:[email-masked]] On Behalf Of 
Haynes, Tom
Sent: Monday, May 20, 2013 9:05 PM
To: Marc Eshel
Cc: nfsv4 list
Subject: Re: [nfsv4] NFSv4.3 [was Re: A more coherent data cache for 
NFS4.3 clients]

---

From: "Noveck, David" <[email-masked]>
Date: Tue, 28 May 2013 16:08:16 -0400

> It was my mistake to call it 4.3.

OK, but in fairness I have to point out that just about everyone seems to make the same sort of mistake.

> Isn't that the process to adding features?

It certainly should be.

Whether it is the process is more difficult to fathom.  Although Nico has used the word "anathema" in describing his sense of the response to a less lock-step approach, I don't know that anyone has actually been anathematized.  I do get the sense that such proposals as yours are looked at askance, at least in some quarters.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Marc Eshel
Sent: Tuesday, May 28, 2013 2:24 PM
To: Noveck, David
Cc: Haynes, Tom; nfsv4 list
Subject: Re: [nfsv4] NFSv4.3 [was Re: A more coherent data cache for NFS4.3 clients]

I think that I already responded to this concern but I will try again. It was my mistake to call it 4.3. At this point it is just future work driven by a personal draft, so the editor of the spec should not be concern about extra work for now. When the draft looks usable and there is an implementation, only than it will be merged to the current version that the group is working on. So why bring it to the list? because I want to make sure that I am working on something of interest to the group that might get accepted when it is ready, I already got good input, like the fact that Trond already started a draft for this work, and that we need posix write semantics. Isn't that the process to adding features?
Marc.

From:        "Noveck, David" <[email-masked]<mailto:[email-masked]>>
To:        "Haynes, Tom" <[email-masked]<mailto:[email-masked]>>, Marc Eshel/Almaden/IBM@IBMUS,
Cc:        nfsv4 list <[email-masked]<mailto:[email-masked]>>
Date:        05/28/2013 09:12 AM
Subject:        RE: [nfsv4] NFSv4.3 [was Re: A more coherent data cache for        NFS4.3        clients]
________________________________

Good serialization:

Bad serialization:

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Monday, May 20, 2013 9:05 PM
To: Marc Eshel
Cc: nfsv4 list
Subject: Re: [nfsv4] NFSv4.3 [was Re: A more coherent data cache for NFS4.3 clients]