Subject: [nfsv4] Fwd: Expiration impending: <draft-ietf-nfsv4-migration-issues-16.txt>

From: David Noveck <[email-masked]>
Date: Mon, 18 Feb 2019 14:36:31 -0500

My understanding is that the woking group is OK with not publishing this as
an Informtional Rfc and just letting it expire.  If there is anyone who has
a problem with that, please raise the issue on this list.   I can renew the
document to allow us time to discuss the possiility of publication, if
there is interest in that.

---------- Forwarded message ---------
From: IETF Secretariat <[email-masked]>
Date: Mon, Feb 18, 2019 at 7:42 AM
Subject: Expiration impending: <draft-ietf-nfsv4-migration-issues-16.txt>
To: <[email-masked]>
Cc: <[email-masked]>, <[email-masked]>

The following draft will expire soon:

Name:     draft-ietf-nfsv4-migration-issues
Title:    NFSv4 Migration and Trunking: Implementation and Specification
Issues
State:    AD is watching
Expires:  2019-02-23 (in 4 days, 19 hours)

---

From: "Marc Eshel" <[email-masked]>
Date: Fri, 22 Feb 2019 15:25:26 -0800

What is the file size returned by the NFS server for getattr after an 
unstable write to the NFS client that did the write or to other NFS 
clients.

As far as I know most file systems will always return the file size that 
includes the unstable writes. 

Does the NFSv4 spec allows the server to return file size that doesn't 
include the unstable write to the writer or any other NFS client?

Marc.

---

From: David Noveck <[email-masked]>
Date: Fri, 22 Feb 2019 18:38:56 -0500

> include the unstable write to the writer or any other NFS client?

I would say "no".   Consider the followiing sentence in the description of
COMMIT.

> If the count is zero, a flush from the offset to the end of the file is
done.

Note that he size returned by GETATTR gives you the end of the file so
that, if it did not
reflect the unstable writes, COMMIT wouldn't work right.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 22 Feb 2019 15:43:28 -0800

> On Feb 22, 2019, at 3:38 PM, David Noveck <[email-masked]> wrote:
> 
> I would say "no".   Consider the followiing sentence in the description of COMMIT.
> 
> > If the count is zero, a flush from the offset to the end of the file is done.
> 
> Note that he size returned by GETATTR gives you the end of the file so that, if it did not
> On Fri, Feb 22, 2019 at 6:25 PM Marc Eshel <[email-masked] <mailto:[email-masked]>> wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

Assume a fresh file, you do a WRITE, and get back the count in the WRITE4resok.

You would expect that to be the same length as a subsequent GETATTR - notice nothing was said about the stable_how4 in the argument.

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Mon, 25 Feb 2019 09:49:37 +0100 (CET)

----- Original Message -----
> From: "Dave Noveck" <[email-masked]>
> To: "Marc Eshel" <[email-masked]>
> Cc: "linux-nfs" <[email-masked]>, "Ganesha-devel" <[email-masked]>, "NFSv4" <[email-masked]>,
> "linux-nfs-owner" <[email-masked]>
> Sent: Saturday, February 23, 2019 12:38:56 AM
> Subject: Re: [nfsv4] file size and getattr

> I would say "no".   Consider the followiing sentence in the description of

I would say "yes". Let's have a look at LAYOUTCOMMIT. Spec says:

"The metadata server may
   use this information to determine whether the file's size needs to be
   updated.  If the metadata server updates the file's size as the
   result of the LAYOUTCOMMIT operation, it must return the new size
   (locr_newsize.ns_size) as part of the results."

I read this as: Metadata server may not know real file size until client sends
the file size information. Our DSes return DATA_SYNC and relay on client to
issue LAYOUTCOMMIT to update file size (well, on close we force data servers
up update file anyway).

Tigran.

---

From: David Noveck <[email-masked]>
Date: Mon, 25 Feb 2019 10:50:21 -0500

Your question was about an NFS server, which I assumed meant an integrated
data server and metadata server.  In that case, the answer is "no".   In
the case in which the data server and metadata server are distinct, there
can be a disagreement but the data server has to know the corect length of
the file even if there is no way for the client to ask for it.  For
example, if you (or another client) read data written unstably, and there
was been no restart you will see bytes past the eof as known by the
metadata server.

BTW, I'm curous whether this poential inconsistency can occur even if the
writes are done stably.  I assume it can since, in that case, you would be
in the same position n which a COMMON, but not a LAYOUTCOMMIT, had been
done.

On Mon, Feb 25, 2019 at 3:49 AM Mkrtchyan, Tigran <[email-masked]>
wrote:

> > Cc: "linux-nfs" <[email-masked]>, "Ganesha-devel" <
> [email-masked]>, "NFSv4" <[email-masked]>,

---

From: Rick Macklem <[email-masked]>
Date: Tue, 26 Feb 2019 02:54:08 +0000

>issue LAYOUTCOMMIT to update file size (well, on close we force data servers

For the pNFS case, when I implemented a pNFS server for FreeBSD, I assumed
that the Metadata server only needed to return a correct size for a file
to a client doing a Getattr with a read/write layout after it did a LayoutCommit.
This implementation worked ok for the FreeBSD client, but did not work
correctly for the Linux-4.17-rc2 kernel.
To fix the server implementation to interoperate with the above Linux kernel,
I had to add a tunable that makes the server get an up to date size for the
Getattr operation for a client when a read/write layout is issued to the client.
Just to be clear, I am referring to the case where the Getattr was done by the
client that holds the read/write layout and not another client.
(This does result in additional overheads whenever a client holds a read/write
 layout for the file.)
I can't remember exactly how the Linux client failed, but I suspect it would
see a premature EOF when the size returned by the MDS was smaller than
the actual size of the file on the DS.

I honestly don't know if this is a bug in the Linux client or a misinterpretation
of RFC5661?

rick

---

From: Rick Macklem <[email-masked]>
Date: Tue, 26 Feb 2019 03:23:42 +0000

I wrote:
[stuff snipped]
>I honestly don't know if this is a bug in the Linux client or a misinterpretation
A little more info on this.

If I recall correctly, the Linux client only did LayoutCommits after Commits.
As such, no LayoutCommit were done after writes to a DS that were FILE_SYNC
(or stable, if you prefer). Without the LayoutCommit, the pNFS server didn't
know when it needed to get an up to date size.
The FreeBSD client does a LayoutCommit after writing to a DS whenever it is done
writing, such as an fsync() or close() syscall on the file or an unlock of a write lock
on the file.

rick

---

From: [email-masked] (J. Bruce Fields)
Date: Tue, 26 Feb 2019 14:46:06 -0500

On Mon, Feb 25, 2019 at 10:50:21AM -0500, David Noveck wrote:
> For example, if you (or another client) read data written unstably,
> and there was been no restart you will see bytes past the eof as known
> by the metadata server.

Possibly worth noting that if there *is* a restart, all bets are off.

And how do you know there hasn't been a restart?  Or if you can figure
that out (e.g. if you're using sessions), how do you know when clients
have finished retransmitting unstable write data?

I'd be curious what prompted the original question.

--b.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 27 Feb 2019 00:13:00 +0000

Trond Myklebust wrote:
[stuff snipped]
> Please see the Errata ID 2751 http://www.rfc-editor.org/errata/eid2751

I'll admit I hadn't seen this errata before. However, it seems to be specific to
the File Layout. For the Flexible File Layout...

When I look in RFC-8435, I cannot find anything that states that a LayoutCommit
is only required for case(s) where a Commit to the Storage Server is required.
Sec. 2.1
   Clearly states that a Commit to the Storage Server is required before the client
   does a LayoutCommit when the write(s) were not done FILE_SYNC.
   However, I do not see any indication that the LayoutCommit is not to be done
   for the case where the write(s) are done FILE_SYNC.

FF_FLAGS_NO_LAYOUTCOMMIT can be used to indicate to a client that
LayoutCommits are not required, but this does not be dependent on how
the write(s) to the Storage Server were done.

The only way a Flexible File layout Metadata server can know what the
current file size is (when a read/write layout is issued to a client) is to do a
Getattr to the Storage Server.
If a client is not required to do a LayoutCommit when the write(s) to the
Storage Server are done FILE_SYNC, then the Metadata server must do
Getattr RPCs to the Storage Server whenever it needs an up to date file size
if a read/write layout is issued to a client.

This can result in a lot of overhead that can be avoided by requiring the
LayoutCommit to be done by a client after writing to a Storage Server,
irrespective of the need for a Commit to the Storage Server.
As such, I would rather not have this errata applied to RFC-8435.

rick

---

From: David Noveck <[email-masked]>
Date: Wed, 27 Feb 2019 12:52:13 -0500

> However note
> that the counter argument to what you state above is that _if_ the
> server requires a layoutcommit before it will acknowledge a file size
> change, then pNFS is likely to underperform for applications such as
> databases or VMs where each record is required to be written in stable
> mode.
> IOW: If all writes that need to be stable are also required to be
> acknowledged with a layoutcommit (to the MDS),

But it is not true that *all* writes that need to be stable are also
required
to be acknowledged with a layoutcommit (to the MDS.  Only those that
potentially change the file size require this.

> then your ability to
> scale out your server will be in doubt

For many applications, particularly databases, it will easy to make sure
that the writes that potentially change the file size are few and far
between.

On Tue, Feb 26, 2019 at 8:12 PM Trond Myklebust <[email-masked]>
wrote:

> On Wed, 2019-02-27 at 00:13 +0000, Rick Macklem wrote:
> > Trond Myklebust wrote:
> > [stuff snipped]
> Fair enough. I agree that the errata in question only applies to the
> pNFS files layout, however you were talking about RFC5661 and whether
> or not we were interpreting that correctly. Since RFC5661 only refers
> to about the behaviour of the pNFS files layout, then I assumed that
> was what you were referring to.
>
> For flexfiles we may have a bug in the layoutcommit case. However note
> that the counter argument to what you state above is that _if_ the
> server requires a layoutcommit before it will acknowledge a file size
> change, then pNFS is likely to underperform for applications such as
> databases or VMs where each record is required to be written in stable
> mode.
> IOW: If all writes that need to be stable are also required to be
> acknowledged with a layoutcommit (to the MDS), then your ability to
> scale out your server will be in doubt.
>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> [email-masked]
>
>
>

---

From: David Noveck <[email-masked]>
Date: Wed, 27 Feb 2019 13:09:03 -0500

> That's true for POSIX O_DSYNC writes, but it is not true for O_SYNC. In
> the latter case, the timestamps are required to be updated
> synchronously as well, which implies a layoutcommit.

Ouch!!

On Wed, Feb 27, 2019 at 1:05 PM Trond Myklebust <[email-masked]>
wrote:

> On Wed, 2019-02-27 at 12:52 -0500, David Noveck wrote:
> > > However note
> That's true for POSIX O_DSYNC writes, but it is not true for O_SYNC. In
> the latter case, the timestamps are required to be updated
> synchronously as well, which implies a layoutcommit.
>
> >
> > > then your ability to
> If the database uses O_DSYNC, yes.
>
> >
> > On Tue, Feb 26, 2019 at 8:12 PM Trond Myklebust <
> > [email-masked]> wrote:
> > > > I'll admit I hadn't seen this errata before. However, it seems to
> > > be
> > > > specific to
> > > > When I look in RFC-8435, I cannot find anything that states that
> > > a
> > > > LayoutCommit
> > > > is only required for case(s) where a Commit to the Storage Server
> > > is
> > > > required.
> > > > Sec. 2.1
> > > >    However, I do not see any indication that the LayoutCommit is
> > > not
> > > > to be done
> > > > current file size is (when a read/write layout is issued to a
> > > client)
> > > > is to do a
> > > > If a client is not required to do a LayoutCommit when the
> > > write(s) to
> > > > the
> > > > Getattr RPCs to the Storage Server whenever it needs an up to
> > > date
> > > > file size
> > > > This can result in a lot of overhead that can be avoided by
> > > requiring
> > > > the

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Wed, 27 Feb 2019 23:13:29 +0100 (CET)

----- Original Message -----
> To: "trondmy" <[email-masked]>
> Cc: "NFSv4" <[email-masked]>, "Tigran Mkrtchyan" <[email-masked]>, "Rick Macklem" <[email-masked]>
> Sent: Wednesday, February 27, 2019 7:09:03 PM

>> synchronously as well, which implies a layoutcommit.

There is no free cheese...In Distributed systems consistency has a high
price tag and the currency measured in milliseconds.

On the other hand, not all applications require O_SYNC. Though, there are of course
some specific `killer` use cases, like cluster-wide checkpointing in HPC.

Tigran.

---

From: Rick Macklem <[email-masked]>
Date: Thu, 28 Feb 2019 01:22:14 +0000

Tigran Mkrtchyan wrote:
[stuff snipped]
>some specific `killer` use cases, like cluster-wide checkpointing in HPC.
Thanks for the interesting discussion everyone.

I don't have a good answer for the O_SYNC case and will be comfortable with
whichever is done for the Flexible File Layout (I suppose some sort of errata for
RFC-8435 might be needed?) w.r.t. this.
(At this time it is a tunable in the FreeBSD pNFS server, but I can't think of an easy
 way for the client to tell the server whether it will be doing LayoutCommits for
 writes with FILE_SYNC, so leaving it as a tunable is all I can think of to do.)

rick