Subject: Re: [nfsv4] layoutcommits and file layout

From: Benny Halevy <[email-masked]>
Date: Mon, 03 Jan 2011 16:21:07 +0200

On 2010-12-17 01:07, Christoph Hellwig wrote:
> On Thu, Dec 16, 2010 at 11:21:21AM -0500, Matt W. Benjamin wrote:
>> Hi,
>>
>> We have a files implementation which wants to receive LAYOUTCOMMIT when a client is finished with a layout.  It was my clear understanding from rfc5661 that we could expect this behavior.
> 
> Care to post it to the list?
> 

I don't know what Matt's server is doing but the fundamental problem is
manifested with extending a file with parallel DS writes.
Assuming that the DS writes are executed in arbitrary order,
exposing the file length before LAYOUTCOMMIT can cause
a concurrent reader to read a hole.  Although locking can
solve this case, day-to-day applications that work well over
local filesystem and legacy NFS may break because of this.

Benny

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 03 Jan 2011 09:40:11 -0500

> local filesystem and legacy NFS may break because of this.

...and this differs from ordinary NFS writes exactly how?

Both cached and uncached (i.e. O_DIRECT) writes can and will be flushed
to disk in entirely random order when writing to the MDS. If you have a
parallel reader on another client (or even on the same client in the
case of O_DIRECT), and want it to see accurate data, then use locking.
If not, you will see holes and other strangeness.

IOW: There are no 'day-to-day applications that work well over legacy
NFS' that rely on this behaviour.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 05 Jan 2011 21:01:24 +0200

> NFS' that rely on this behaviour.

Assuming the client writes sequentially (over tcp) the writes will
practically be processed in order into the server's cache so with
no crashes in the mix a parallel reader will see no holes.
I'd really like the following scenario to work over pNFS with
no hassles:
	"some app >> foo" on one client, and
	"tail -f foo" on another

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 05 Jan 2011 14:04:23 -0500

> 	"some app >> foo" on one client, and

No, that doesn't work today! Believe me, I get the "bug reports"...

There is no point in trying to add properties to pNFS that don't exist
with ordinary NFS.

Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 05 Jan 2011 14:14:34 -0500

> There is no point in trying to add properties to pNFS that don't exist

...and for the record: use of TCP does _not_ suffice to ensure writes
are processed in order.

In the Linux kernel, we have all sorts of parallelism going on before
the writes even hit the socket on the client. Everything from background
flushing to queuing in the sunrpc layer (e.g. for a session slot)
conspires to destroy any hope of ever achieving what you propose above.

That's not even counting what goes on with the server side. Think, for
instance, of the case where the server crashes before a COMMIT has been
successfully sent. Not only will your reader see holes, it will think
the file has been truncated...

Trond

NetApp