Subject: [nfsv4] Progressing RFC errata for RFC 5661

From: Magnus Westerlund <[email-masked]>
Date: Fri, 13 Sep 2019 08:27:01 +0000

Hi,

To make progress on two aspects, both ensure that the RFC errata are dealt
with, and to progress draft-ietf-nfsv4-rfc5661sesqui-msns-01 I like to do
the following:

>From the discussion it appears that some think that the reported errata
should mostly be accepted. Based on that I want at least one WG participant
to look at each errata and public state their opinion on what Status that
particular errata should be in. Before doing that please read
https://www.rfc-editor.org/errata-definitions/ and the IESG statement on
errata
https://www.ietf.org/about/groups/iesg/statements/processing-rfc-errata/ . 

Below you have the list of errata in reported state. When you have looked at
one and judged, you respond to this email and write below the specific
errata your recommendation. I will then review that errata to see if I
concur with the judgment and then make it so. 

RFC Number
(Errata ID)

Section

Type

Source of RFC <https://www.rfc-editor.org/source/> 

Submitted By

Publication Format

Date Submitted

RFC5661 (2505 <https://www.rfc-editor.org/verify_errata_select.php?eid=2505>
)

12.5.4.

Technical

nfsv4 (tsv)

Michael Eisler

TEXT

2010-08-31

RFC5661 (2751 <https://www.rfc-editor.org/verify_errata_select.php?eid=2751>
)

GLOBAL

Technical

nfsv4 (tsv)

Ricardo Labiaga

TEXT

2011-03-21

RFC5661 (3379 <https://www.rfc-editor.org/verify_errata_select.php?eid=3379>
)

18.50.3

Technical

nfsv4 (tsv)

Asmita Karandikar

TEXT

2012-10-15

RFC5661 (3714 <https://www.rfc-editor.org/verify_errata_select.php?eid=3714>
)

13.4.4.

Technical

nfsv4 (tsv)

Yuri Radchenko

TEXT

2013-08-31

RFC5661 (3901 <https://www.rfc-editor.org/verify_errata_select.php?eid=3901>
)

18.43.3

Technical

nfsv4 (tsv)

Trond Myklebust

TEXT

2014-02-25

RFC5661 (4119 <https://www.rfc-editor.org/verify_errata_select.php?eid=4119>
)

12.2.10

Technical

nfsv4 (tsv)

Christoph Hellwig

TEXT

2014-09-17

RFC5661 (4492 <https://www.rfc-editor.org/verify_errata_select.php?eid=4492>
)

18.17.4

Technical

nfsv4 (tsv)

David Noveck

TEXT

2015-10-05

RFC5661 (4711 <https://www.rfc-editor.org/verify_errata_select.php?eid=4711>
)

15.1.5.5

Technical

nfsv4 (tsv)

Tom Haynes

TEXT

2016-06-16

RFC5661 (4712 <https://www.rfc-editor.org/verify_errata_select.php?eid=4712>
)

6.2.1.3.1.

Technical

nfsv4 (tsv)

Trond Myklebust

TEXT

2016-06-16

RFC5661 (5040 <https://www.rfc-editor.org/verify_errata_select.php?eid=5040>
)

18.46.3

Technical

nfsv4 (tsv)

Jonathan Price

TEXT

2017-06-12

RFC5661 (5212 <https://www.rfc-editor.org/verify_errata_select.php?eid=5212>
)

15.2

Technical

nfsv4 (tsv)

NFS4ERR_ROFS is not a valid error code for LAYOUTGET

TEXT

2017-12-19

RFC5661 (4572 <https://www.rfc-editor.org/verify_errata_select.php?eid=4572>
)

18.35.3

Editorial

nfsv4 (tsv)

Sai Chakravarthy Tangudu

TEXT

2015-12-30

RFC5661 (4914 <https://www.rfc-editor.org/verify_errata_select.php?eid=4914>
)

14.3.3

Editorial

nfsv4 (tsv)

Dylan Simon

TEXT

2017-01-22

RFC5661 (5476 <https://www.rfc-editor.org/verify_errata_select.php?eid=5476>
)

18.42.3

Editorial

nfsv4 (tsv)

Tigran Mkrtchyan

TEXT

2018-08-23

Cheers

Magnus Westerlund

---

From: David Noveck <[email-masked]>
Date: Sun, 15 Sep 2019 09:44:25 -0400

> with, and to progress draft-ietf-nfsv4-rfc5661sesqui-msns-01 I like to do
Since there appear to be no errata that concern changes made in
rfc5661sesqui, I believe the real benefit will be in making sure we
understand our errrata as they relate to the eventual publication of
rfc5661bis.

> should mostly be accepted.
>

True, but once we discuss some of these in detail, there will be
differences of opinions to resolve.  Also, acceptance of the essence of the
change in rfc5661bis is compatible with multiple errata statuses.

> Based on that I want at least one WG participant to look at each errata
> and public state their opinion on what Status that particular errata should
> be in.
>

I'll do that, but I will also state what we want to do with the errata in
an eventual rfc5661bis.  This is separate from the errata status although
related to it.  The possibilities are to accept it as-is, accept it with
modifications, or to accept it after it is confirmed by working group
discussions. There might be a few that merit immediate rejection although I
haven't seen any.

Even the one errata in status *Rejected* should be incorporated into
rfc5661bis, although the current status is appropriate given current IETF
rules. Errata 2751 is similar.

> Before doing that please read
I did read that but feel that, for many errata, the official status may not
reflect what we evenually do.   Still, I will try to provide a reasonable
official status in line with these guidelines, to the extent one can be
determined.

>
>
> Below you have the list of errata in reported state. When you have looked
> at one and judged, you respond to this email and write below the specific
As I understand it, you'll deal with the official status and the working
group will, in many cases, have further discssions about what should be
done in rfc5661bis.  Of course, any member of the working group could
recommend a different status if they feel it is appropriate.

I've provided recommendations for the first seven errata.  I hope others
will be able to pitch in soon to help complete this work.

> *RFC Number(Errata ID)*
>
> *Section*
>
> *Type*
>
> *Source of RFC <https://www.rfc-editor.org/source/>*
>
> *Submitted By*
>
> *Publication Format*
>
> *Date Submitted*
>
> RFC5661 (2505
> <https://www.rfc-editor.org/verify_errata_select.php?eid=2505>)
>
> 12.5.4.
>
> Technical
>
> nfsv4 (tsv)
>
> Michael Eisler
>
> TEXT
>
> 2010-08-31
>

Recommend this be  assigned status* Hold for Document Update* since it
*might* (emphasis added) change an existing consensus decision.

The working group needs to discuss these changes before we decide to
incorporate them in rfc5661bis.

An additional issue that needs to be addressed before incorporating these
changes is that it is not appropriate to add extensive material about
handling of the file layout to Section 12.  That belongs in Section 13. We
don't want to exacerbate the confusion that Tom has noted between
statements about layouts in general and  about file layouts.  Rfc5661bis
needs to do better in this respect.

> RFC5661 (2751
> <https://www.rfc-editor.org/verify_errata_select.php?eid=2751>)
>
> GLOBAL
>
> Technical
>
> nfsv4 (tsv)
>
> Ricardo Labiaga
>

This is similar to the errata above although, because of the extensiveness
of the changes proposed I think *Rejected* is more appropriate as an errata
status.

The working group needs to discuss these changes before going forward with
them. Also, any new section regarding the file layout should appear within
Section 13.

>
>
> RFC5661 (3379
> <https://www.rfc-editor.org/verify_errata_select.php?eid=3379>)
>
> 18.50.3
>
> Technical
>
> nfsv4 (tsv)
>
> Asmita Karandikar
>
> TEXT
>
> 2012-10-15
>
Recommend this be assigned status *Verified* and incorporated into
rfc5661bis as-is.

> TEXT

Recommend this be assigned status *Hold For Document Update (*based on the
fact that the lck o thes changes isunlikely to result in difficulty) and
incorporated in rfc5661bis as-is.

> 2014-02-26

Recommend this be assigned status *Hold For Document Update* based on the
possibility that it could affect  (whether justifiably or not) an existing
consensus decision.

This should be discussed by the working group together with related pNFS
errata.  A good time to do that is when we have a full draft replacement
for Sections 12 and 13, incorporating Tom's work in rfc8434.

BTW, the report refers to errata 3226, which doesn't seem to be obtainable.
In any case, some re-organization of pNFS errata seems to be necessary
including the 5 currently in status *Hold for Document Update *dealing with
LAYOUTCOMMIT, before we can incorporate these in a draft rfc5661bis.

>
> RFC5661 (4119
> <https://www.rfc-editor.org/verify_errata_select.php?eid=4119>)
>
> 12.2.10
>
> Technical
>
> nfsv4 (tsv)
>
> Christoph Hellwig
>
> TEXT
>
> 2014-09-17
>

Recommend this be assigned status *Hold For Document Update* based on the
possibility that this affects a consensus decision.  Whether it does so
depends on whether you consider it possible for the working group to arrive
at two contradictory consensus decisions as may have happened here with
regard to device lifetime.  If that is possible, then this errata
contradicts at least one of those.  On the other hand, one might argue that
such a contradiction reflects not consensus but unacknowkedged confusion so
that the errata represents necessary clarification of a consensus
non-decision rather than an attempt to change a consensus decision. Sigh.

In any case, the working group needs to consider these proposed changes,
together with related pNFS issues. It might adopt this approach or try to
address the underlying contradictions in a different way.

> RFC5661 (4492
> <https://www.rfc-editor.org/verify_errata_select.php?eid=4492>)
>
> 18.17.4
>
> Technical
>
> nfsv4 (tsv)
>
> David Noveck
>
> TEXT
>
> 2015-10-05
>

Recommend this be assigned status *Verified* and incorporated in rfc5661bis
as-is.

Of course, I might be biased :-)

> )
>
> 15.1.5.5
>
> Technical
>
> nfsv4 (tsv)
>
> Tom Haynes
>
> TEXT
>
> 2016-06-16
>
> RFC5661 (4712
> <https://www.rfc-editor.org/verify_errata_select.php?eid=4712>)
>
> 6.2.1.3.1.
>
> Technical
>
> nfsv4 (tsv)
>
> Trond Myklebust
>
> TEXT
>
> 2016-06-16
>
> RFC5661 (5040
> <https://www.rfc-editor.org/verify_errata_select.php?eid=5040>)
>
> 18.46.3
>
> Technical
>
> nfsv4 (tsv)
>
> Jonathan Price
>
> TEXT
>
> 2017-06-12
>
> RFC5661 (5212
> <https://www.rfc-editor.org/verify_errata_select.php?eid=5212>)
>
> 15.2
>
> Technical
>
> nfsv4 (tsv)
>
> RFC5661 (4572
> <https://www.rfc-editor.org/verify_errata_select.php?eid=4572>)
>
> 18.35.3
>
> Editorial
>
> nfsv4 (tsv)
>
> Sai Chakravarthy Tangudu
>
> TEXT
>
> 2015-12-30
>
> RFC5661 (4914
> <https://www.rfc-editor.org/verify_errata_select.php?eid=4914>)
>
> 14.3.3
>
> Editorial
>
> nfsv4 (tsv)
>
> Dylan Simon
>
> TEXT
>
> 2017-01-22
>
> RFC5661 (5476
> <https://www.rfc-editor.org/verify_errata_select.php?eid=5476>)
>
> 18.42.3
>
> Editorial
>
> nfsv4 (tsv)
>
> Tigran Mkrtchyan
>
> TEXT
>
> 2018-08-23
>
>
>
> Cheers
>
>
>
> Magnus Westerlund
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 16 Sep 2019 16:26:12 -0400

On Sun, 15 Sep 2019 at 09:44, David Noveck <[email-masked]> wrote:

> of the changes proposed I think *Rejected* is more appropriate as an
> errata status.
>

Rejecting this would break clients that have been encoded to rely on that
behaviour (i.e the Linux client, for one). Without it, we would be unable
to determine whether or not a LAYOUTCOMMIT is required for the files pNFS
layout type, and would have to default to assuming it does. That would put
the existing client base in violation. It would also force us to start
sending LAYOUTCOMMIT to servers that currently do not require it (i.e.
NetApp).

I'm not aware of any pNFS files server implementation that is in violation
of this errata and that is expecting a LAYOUTCOMMIT on a commit-through-MDS
implementation, or following a FILE_SYNC return from a commit-through-DS
implementation. Ccing Rick and Tigran for their input.

---

From: David Noveck <[email-masked]>
Date: Mon, 16 Sep 2019 17:29:17 -0400

On Mon, Sep 16, 2019 at 4:26 PM Trond Myklebust <[email-masked]> wrote:

>> This is similar to the errata above although, because of the
>> extensiveness of the changes proposed I think *Rejected* is more
>> appropriate as an errata status.
>>
>
> behaviour (i.e the Linux client, for one).
>

Rejecting the errata (i.e. putting it in the state "Rejected") would not do
that.

What would do that is doing an rfc5661bis that does not incorporate these
changes, but I don't think the working group is going to do that, for the
reasons you state.

> Without it, we would be unable to determine whether or not a LAYOUTCOMMIT
> is required for the files pNFS layout type, and would have to default to
> assuming it does. That would put the existing client base in violation.
>

It appears that the existing client base is in violation of rfc5661.   I
don't think we want rfc5661bis to be the same, so I expect the working
group will decide to make this sort of change.

It would also force us to start sending LAYOUTCOMMIT to servers that
> currently do not require it (i.e. NetApp).
>

If there are such server implementations, people will be able to tell us
about them, when we review the rfc5661bis section replacing Section 13 of
rfc5661.

The basic issue is that errata are for correcting situations in which an
RFC does not correctly represent the decisions of the working group.  I
believe that, in this case, the working group wants to change its decision,
for the good reasons you have indicated.   By saying that the errata should
be put in status "Rejected", I do not mean to imply that these changes
should not go into rfc5661bis.   I'm only saying that, according to the
IETF rules that Magnus pointed us to, this is the correct status.   Magnus
will make the final decision on errata status.   Regarding the
incorporation of these changes into rfc5661bis, the working group will make
a decision which the editor will then try to sell to the IESG, using the
kind of arguments that you have made above.   The IESG does not want the
spec to diverge from exsting implementations, so I don't expect a lot of
pushback, once the situation is made clear to them.

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Mon, 16 Sep 2019 23:50:00 +0200 (CEST)

----- Original Message -----
> From: "Dave Noveck" <[email-masked]>
> To: "Trond Myklebust" <[email-masked]>
> Cc: "Tigran Mkrtchyan" <[email-masked]>, "Rick Macklem" <[email-masked]>, "Magnus Westerlund"
> <[email-masked]>, "NFSv4" <[email-masked]>
> Sent: Monday, September 16, 2019 11:29:17 PM
> Subject: Re: [nfsv4] Progressing RFC errata for RFC 5661

> about them, when we review the rfc5661bis section replacing Section 13 of

If I read suggested errata correctly, then our server does exactly what is suggested:

We do not set NFL4_UFLG_COMMIT_THRU_MDS flag and return data server returns
DATA_SYNC4 on WRITE, with expectation is that client will send a LAYOUTCOMMIT.

> pushback, once the situation is made clear to them.

This diverge is caused by 'on the field' experience, which covered by rfc 5661
not 100% precise.

Thanks,
   Tigran.

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 17 Sep 2019 08:41:48 +0000

Hi,

Looking at Errata 2751: 
https://www.rfc-editor.org/errata_search.php?eid=2751

I think it is fairly obvious that Approving this would not be a
clarification of what RFC5661 descibes. It appears to me (with my lack
of good understanding of NFS) to be a change of the consensus behavior
that was established when RFC 5661 was published. 

That the WG consensus today may be to accept the text in to RFC 5661bis
is not relevant and that the implementation follows this. The way to
resolve this is to establish the consesus for implement these changes
into RFC 5661bis and then publish it replacing RFC 5661. 

Cheers

Magnus

On Mon, 2019-09-16 at 23:50 +0200, Mkrtchyan, Tigran wrote:
> > Cc: "Tigran Mkrtchyan" <[email-masked]>, "Rick Macklem" <
> > [email-masked]>, "Magnus Westerlund"
> > > > > <
> > > > > https://protect2.fireeye.com/url?k=0f81077e-53130a61-0f8147e5-0cc47ad93db4-61f6126ac10042d2&q=1&u=https%3A%2F%2Fwww.rfc-editor.org%2Fverify_errata_select.php%3Feid%3D2751
> > > > > >)
> > > > > 
> > > > > GLOBAL
> > > > > 
> > > > > Technical
> > > > > 
> > > > > nfsv4 (tsv)
> > > > > 
> > > > > Ricardo Labiaga
> > > > > 
> > > > 
-- 
Cheers

Magnus Westerlund 

----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 17 Sep 2019 07:35:33 -0400

At the time when this was raised to the IETF WG, we argued that it is
actually a clarification because the original RFC5661 does not specify
how to handle the different types of WRITE persistence.
IOW: it is trying to clarify an omitted description of client/server behaviour.

There is no text in RFC5661 that states what the behaviour of a pNFS
client should be when the server returns FILE_SYNC in response to a
WRITE (as NetApp servers always do). The DESCRIPTION in section
18.32.3 for a standard WRITE through the MDS states that means all
data+metadata has been persisted, but what does that even mean for
pNFS, where the data and metadata are handled by different servers?
You can't just port the description in 18.32.3, because it makes no
sense in the context of the different MDS/DS roles.

The errata presented by Ricardo therefore represents the consensus
interpretation for dealing with the question of what the different
WRITE modes mean for the 'files' pNFS type, and how they relate to
COMMIT and LAYOUTCOMMIT.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 18 Sep 2019 03:27:29 +0000

Trond Myklebust wrote:
>On Sun, 15 Sep 2019 at 09:44, David Noveck ><[email-masked]<mailto:[email-masked]>> wrote:
>>
>>An additional issue that needs to be addressed before incorporating these >>changes is that it is not appropriate to add extensive material about handling >>the file layout to Section 12.  That belongs in Section 13. We don't want to >>exacerbate the confusion that Tom has noted between statements about >>layouts in general and  about file layouts.  Rfc5661bis needs to do better in >>respect.
>>RFC5661 (2751<https://www.rfc-editor.org/verify_errata_select.php?eid=2751>)
>>GLOBAL
>>Technical
>>nfsv4 (tsv)
>>Ricardo Labiaga
>>
>>This is similar to the errata above although, because of the extensiveness of the 
>>changes proposed I think Rejected is more appropriate as an errata status.
>
>Rejecting this would break clients that have been encoded to rely on that >behaviour (i.e the Linux client, for one). Without it, we would be unable to >determine whether or not a LAYOUTCOMMIT is required for the files pNFS >layout type, and would have to default to assuming it does. That would put the >existing client base in violation. It would also force us to start sending >LAYOUTCOMMIT to servers that currently do not require it (i.e. NetApp).
Just fyi, this is what the FreeBSD client does. It always sends a Layoutcommit
to the MDS whenever an fsync(2) or similar is done and the client holds
a write layout for the file, unless the server replies NFS4ERR_NOTSUPP to Layoutcommit. (I think that the Netapp Filer replies NFS4ERR_NOTSUPP to Layoutcommit, which was why it disables doing Layoutcommits on a mount
once it sees a NFS4ERR_NOTSUPP reply.)
--> Layoutcommits are done even if the DS replies FILE_SYNC and/or the MDS
       sets commit-through-MDS.

>I'm not aware of any pNFS files server implementation that is in violation of this >errata and that is expecting a LAYOUTCOMMIT on a commit-through-MDS >implementation, or following a FILE_SYNC return from a commit-through-DS >implementation. Ccing Rick and Tigran for their input.
Well, when I first wrote the pNFS server, I assumed that the Layoutcommit
would be done and used it to synchronize the FileSize, Change attributes
between the DS and the ones cached in the MDS.

Then, I found out this didn't work for the Linux client, so I added a tunable
sysctl called vfs.nfsd.pnfsgetdsattr which tells the server to not cache
FileSize, Change when a write layout is issued to a client for the file.
As such, when this tunable is set (and that is the default), it works for the
Linux client, but is less efficient, since it must acquire FileSize, Change from
the DS whenever a Getattr requests them, if a write layout has been issued
for the file.
--> The FreeBSD server can be run either assuming the Layoutcommit is always
       done, or if the clients don't do them. In the latter case, it should work
       correctly even if a client never does a Layoutcommit.

I have not seen the wording of the actual errata (the links in these posts
require a login I don't have and I probably missed it when it was emailed),
but I am assuming that it says Layoutcommits should be done by the client
when the DS replies Unstable to writes and the MDS requires
commit-through-DS.

rick

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Wed, 18 Sep 2019 13:06:17 +0200 (CEST)

----- Original Message -----
> From: "Rick Macklem" <[email-masked]>
> To: "Trond Myklebust" <[email-masked]>, "Dave Noveck" <[email-masked]>
> Cc: "Magnus Westerlund" <[email-masked]>, "NFSv4" <[email-masked]>, "Tigran Mkrtchyan"
> <[email-masked]>
> Sent: Wednesday, September 18, 2019 5:27:29 AM

> Trond Myklebust wrote:
>>On Sun, 15 Sep 2019 at 09:44, David Noveck
>>><[email-masked]<mailto:[email-masked]>> wrote:
>>>
>>>handling >>the file layout to Section 12.  That belongs in Section 13. We don't
>>>want to >>exacerbate the confusion that Tom has noted between statements about
>>>>>layouts in general and  about file layouts.  Rfc5661bis needs to do better in
>>>>>respect.
>>>behaviour (i.e the Linux client, for one). Without it, we would be unable to
>>>determine whether or not a LAYOUTCOMMIT is required for the files pNFS >layout
>>type, and would have to default to assuming it does. That would put the
>>>existing client base in violation. It would also force us to start sending
>>>LAYOUTCOMMIT to servers that currently do not require it (i.e. NetApp).
> a write layout for the file, unless the server replies NFS4ERR_NOTSUPP to
> Layoutcommit. (I think that the Netapp Filer replies NFS4ERR_NOTSUPP to
> Layoutcommit, which was why it disables doing Layoutcommits on a mount
>>I'm not aware of any pNFS files server implementation that is in violation of
>>this >errata and that is expecting a LAYOUTCOMMIT on a commit-through-MDS

Hi Rick,

here is the public link to errata

https://www.rfc-editor.org/errata/eid2751

Tigran.

---

From: Rick Macklem <[email-masked]>
Date: Wed, 18 Sep 2019 16:39:21 +0000

Mkrtchyan, Tigran wrote:
[stuff snipped]
>https://www.rfc-editor.org/errata/eid2751
Thanks Tigran.

Ok, so now that I've read it I have to admit I think it is rewriting the RFC
to conform with what the Linux client does.

I think this para. from Sec. 13.10 of RFC-5661 is clear:
   The NFSv4.1 protocol only provides close-to-open file data cache
   semantics; meaning that when the file is closed, all modified data is
   written to the server.  When a subsequent OPEN of the file is done,
   the change attribute is inspected for a difference from a cached
   value for the change attribute.  For the case above, this means that
   a LAYOUTCOMMIT will be done at close (along with the data WRITEs) and
   will update the file's size and change attribute.  Access from
   another client after that point will result in the appropriate size
   being returned.

It states "will be done". It doesn't say anything about UNSTABLE4 vs FILE_SYNC4.
(I think most POSIX-like clients would consider the fsync(2) syscall to require
 the same treatment as "close" above, but that is a POSIX-specific client issue.)
I can see the argument that, since there is no "must" in the statement, that a
client can choose not to do this, but that would also imply that the client will
need to live with the consequences of it.

I think the second sentence of the first para. of the errata is bogus:
For file layouts, WRITEs to a Data Server that return a stable_how4 value of 
FILE_SYNC4 guarantee that data and file system metadata are on stable 
storage.  This means that a LAYOUTCOMMIT is not needed in order to make the 
data and metadata visible to the metadata server and other clients.

Why?
The FILE_SYNC4 was returned by the DS. This would imply the DS
has committed data and metadata to stable storage on the DS.
However, I am not aware of anything in RFC-5661 that would imply that
the Size, Time_Modify and Change attributes or anything else must have
been updated or in stable storage on the MDS at this time.

If a server does not require LayoutCommit operations for correct behaviour
then it can simply reply NFS4ERR_NOTSUPP (as I believe the Netapp filer
does) and the client then no longer needs to do them.

If there is somewhere in RFC-5661 that it is stated that LayoutCommits are
not required when the DS replies FILE_SYNC4, then I missed it and there
is a problem with RFC-5661 that needs to be addressed.

Otherwise, sorry, but it seems that the bug is in the Linux client implementation
and not RFC-5661.

Is there a File Layout pNFS server implementation where the DSs return
FILE_SYNC4 that will break if the client does a LayoutCommit for this case?
(If so, then something may need to be done.)

rick

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 18 Sep 2019 15:58:35 -0400

Rick,

That errata predates most of the Linux pNFS client implementation. We
wrote the implementation to conform to the errata.

So no. It's not a bug. It's a deliberate design based on a decision
that was discussed in the IETF WG, on the mailing list

https://mailarchive.ietf.org/arch/msg/nfsv4/_KTtO6uz-MvRoStbhPuOXWZr6yI

and in a special session of the IETF:

https://mailarchive.ietf.org/arch/msg/nfsv4/Rpw9XCwCARxfaU4ym5L2TauV6ao

---

From: Rick Macklem <[email-masked]>
Date: Thu, 19 Sep 2019 04:07:32 +0000

>https://mailarchive.ietf.org/arch/msg/nfsv4/_KTtO6uz-MvRoStbhPuOXWZr6yI
This actually appears to be a discussion related to the offset and length arguments for LayoutCommit, but...
>https://mailarchive.ietf.org/arch/msg/nfsv4/Rpw9XCwCARxfaU4ym5L2TauV6ao
Ok. This was long before I got around to implementing it, so I wouldn't have
understood the implications.
--> I would have been interested in hearing the rationale behind not doing
      LayoutCommit for FILE_SYNC4 writes, since it seems to me that RFC-5661
      had gotten it right when it required them.

As I said, the FreeBSD server can handle this case, it just results in a lot of
overhead synchronizing Size, Change, Time_Modify between MDS and DS
whenever a RW layout is issued to a client for the file.

Thanks for pointing this out, rick

---

From: "Mkrtchyan, Tigran" <[email-masked]>
Date: Thu, 19 Sep 2019 11:14:37 +0200 (CEST)

----- Original Message -----
> Cc: "Tigran Mkrtchyan" <[email-masked]>, "Dave Noveck" <[email-masked]>, "Magnus Westerlund"
> Sent: Thursday, September 19, 2019 6:07:32 AM

>      had gotten it right when it required them.

In case of a cluster filesystem back-end, FILE_SYNC4 on write indicates that MDS already
has the correct file attributes. An extra LAYOUTCOMMIT will introduce additional overhead.
I can imagine a HPC workload where client talks to DS over InfiniBand and Ethernet to
MDS. An extra LAYOUTCOMMIT will drop write throughput.

Tigran.

---

From: David Noveck <[email-masked]>
Date: Thu, 19 Sep 2019 06:17:17 -0400

>This actually appears to be a discussion related to the offset and length
arguments for LayoutCommit

True.   It appears that this change (essentially eliminating the use of
these arguments) was turned into *5* Held-over errata that just sat there
for over eight years, with not very many people knowing about them and
rfc5661 left describing a protocol thatsome people in the working group
felt to have been superseded.

The case of the meaning of  FILE_SYNC4 in pnfs-file was similar,
except that there was only one errata and it was left in Reported state.

In both cases, we will need information on the rationale for the change,
since the errata mechanism is not designed to record rationales. It appears
that it was presumed that changes big enough to require rationales are too
big for the errata mechanism.   The need for such rationales is not only to
satisfy your curiosity (and mine; it turns out I was in the hospital
while much of this discussion was going on).   These changes will have to
go into rfc5661bis which will need a "changes since rfc5661" section, which
will need, for each of these changes, a short paragraph explaining why the
change was made.

Even more important, if there are other errata that have already been
essentially approved by the working group, as the six we are talking about
have been, we need to hear about them right away, so they can be taken
account of as we develop plans to actually write and publish rfc5661bis.
We need to do this as soon as we can.   Given what we have just found out,
it appears that we are already eight years late in getting started on this
effort.

---

From: Magnus Westerlund <[email-masked]>
Date: Thu, 19 Sep 2019 12:41:04 +0000

Hi,

The reasons I do state that this Errata, despite its importance ends up
in the type of reject are two fold. First of all as it adds new text
for something that didn't exist, i.e. no WG consensus was established
at the point of publication. Secondly, the errata is quite significant.

Therefore, despite this being important it goes way beyond what the
Errata is intended for. The right way forward according to the
established process is to reject the errata and include the changes in
an RFC5661bis document. 

Cheers

Magnus

> <[email-masked]> wrote:
https://protect2.fireeye.com/url?k=07374a3a-5be3435c-07370aa1-8610d8a762ca-0924dae3e4722146&q=1&u=https%3A%2F%2Fwww.rfc-editor.org%2Ferrata_search.php%3Feid%3D2751
> > > > On Mon, Sep 16, 2019 at 4:26 PM Trond Myklebust <
> > > > [email-masked]>
> > > > wrote:
> > > > 
> > > > > 
> > > > > 
> > > > > On Sun, 15 Sep 2019 at 09:44, David Noveck <
> > > > > [email-masked]>
> > > > > wrote:
> > > > > 
> > > > > > 
> > > > > > changes is that it is not appropriate to add extensive
> > > > > > material
> > > > > > about
> > > > > Rejecting this would break clients that have been encoded to
> > > > > rely
> > > > > on that
> > > > What would do that is doing an rfc5661bis that does not
> > > > incorporate
> > > > these
> > > > changes, but I don't think the working group is going to do
> > > > that,
> > > > for the
> > > > reasons you state.
> > > > 
> > > > 
> > > > > I'm not aware of any pNFS files server implementation that is
> > > > > in
> > > > > violation
> > > > > of this errata and that is expecting a LAYOUTCOMMIT on a
> > > > > commit-
> > > > > through-MDS
> > > > > implementation, or following a FILE_SYNC return from a
> > > > > commit-
> > > > > through-DS
> > > > If there are such server implementations, people will be able
> > > > to
> > > > tell us
> > > > about them, when we review the rfc5661bis section replacing
> > > > Section
> > > > 13 of
> > > > rfc5661.
> > > 
> > > If I read suggested errata correctly, then our server does
> > > exactly
> > > what is suggested:
> > > 
> > > We do not set NFL4_UFLG_COMMIT_THRU_MDS flag and return data
> > > server
> > > returns
> > > > believe that, in this case, the working group wants to change
> > > > its
> > > > decision,
> > > > should not go into rfc5661bis.   I'm only saying that,
> > > > according to
> > > > the
> > > > incorporation of these changes into rfc5661bis, the working
> > > > group
> > > > will make
> > > > spec to diverge from exsting implementations, so I don't expect
> > > > a
> > > > lot of
> > > This diverge is caused by 'on the field' experience, which
> > > covered by
> > > rfc 5661
> > > not 100% precise.
> > > 
> > > Thanks,
> > >    Tigran.
> > 
> > --
> > Cheers
> > 
> > Magnus Westerlund
> > 
> > 
-- 
Cheers

Magnus Westerlund

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 19 Sep 2019 10:57:17 -0400

> This actually appears to be a discussion related to the offset and length arguments for LayoutCommit, but...

See Spencer's email:
https://mailarchive.ietf.org/arch/msg/nfsv4/tj4_tBSTLMEKqf43Wg_bv1K0TeI

It may have focussed on the offset and length stuff, but it was
intended to cover all the layoutcommit issues that were being
discussed at the time.

>       had gotten it right when it required them.

It doesn't mention anything about FILE_SYNC4 in the context of
pNFS/files. All it says is that the server is indicating that the data
+ metadata changes have been persisted. Since the MDS/DS back end
protocol is not part of the pNFS/files protocol specification, but is
required to provide strong coupling of NFSv4.1 state, etc between the
MDS and DS, it allows the server to implement strong policies w.r.t.
data+metadata persistence. Since indeed the only server implementation
available at the time actually did persist data + metadata when it
returned FILE_SYNC4, then we needed that clarification.

Note that pNFS/flexfiles does not impose the same interpretation of
the WRITE flags, because the MDS/DS back end protocol is loosely
coupled (and is designed so that NFS can act as that back end
protocol).

Also note that the requirements of POSIX when it comes to O_SYNC
write()s, provides additional motivation here. O_SYNC requires both
data+metadata to be persisted when the write() system call returns.
You cannot get scale out behaviour if each O_SYNC write requires both
I/O to the DS, and then a LAYOUTCOMMIT to the MDS in order to persist
the file metadata. In fact, you can't even get basic NFSv4 protocol
consistency if you allow the file data to visibly change on the DS but
don't record that fact in the change attribute on the MDS. The generic
pNFS spec in RFC5661 section 12.5.4. is extremely oily about this
scenario and seems to put the onus on the clients for ensuring
data/metadata consistency. However it does not at all discuss what
should happen in the case where the client does not do this, or is
incapable of doing so due to a reboot, network partition or crash
other than to note in section 12.7.1. that some data may be lost (Yes,
but what about the metadata, Jim?).

---

From: Rick Macklem <[email-masked]>
Date: Thu, 19 Sep 2019 16:39:08 +0000

Mkrtchyan, Tigran wrote:
>> arguments for LayoutCommit, but...
>>>
>In case of a cluster filesystem back-end, FILE_SYNC4 on write indicates that >MDS already
>has the correct file attributes. An extra LAYOUTCOMMIT will introduce >additional overhead.
>I can imagine a HPC workload where client talks to DS over InfiniBand and >Ethernet to
Ok, I'll assume you have a server which needs LayoutCommit for UNSTABLE4
writes, but doesn't need one for FILE_SYNC4 writes.
(If the server never needs LayoutCommit, it can simply do what I believe
 the Netapp filer does, which is reply NFS4ERR_NOTSUPP.)

I agree that this could result in extra overhead, but at least for some cases
the client could put the LayoutCommit in the same compound as something
like Close (or Commit if the server does commit-through-mds), which would
avoid an extra RPC RTT.
Presumably the server would know it didn't need to do anything and could
just reply NFS_OK for the operation in the FILE_SYNC4 case?

But, yes, this is a case where using the Errata might improve performance.

For NFSv4.2, I think a new recommended attribute could be added, so that
a server could indicate to a client when it wanted a LayoutCommit.
(I think the NFSv4.2 versioning rules would allow this to be added?)
I'm not sure how the client would "handshake" with the server, acknowledging
that it understood the new attribute, though?
It is tempting to add NFL4_UFLG_FILE_SYNC4_LAYOUT_COMMIT, but that
would probably cause problems for extant implementations that don't
expect the flag bit (returning an error instead of ignoring it, etc).

rick

---

From: Rick Macklem <[email-masked]>
Date: Thu, 19 Sep 2019 16:51:13 +0000

Rick Macklem wrote:
>>In case of a cluster filesystem back-end, FILE_SYNC4 on write indicates that >>MDS already
>>has the correct file attributes. An extra LAYOUTCOMMIT will introduce >>additional overhead.
>>I can imagine a HPC workload where client talks to DS over InfiniBand and >>Ethernet to
Duh. I realized that the "handshake" would simply be the client getting the
attribute.

rick