Subject: Re: [nfsv4] RFC 1813 - NFS questions on write verifier

From: "Noveck, David" <[email-masked]>
Date: Thu, 21 Jun 2018 11:37:47 +0000

First of all, I've cc'd the v4 working group, which is a good idea in general when you ask questions about these specs, especially since most of the email addresses on your to-list are probably no longer active (e.g. anything ending eng.sun.com).

  *   RFC 7530 is bit more clear,

I think it is less clear.  RFC1813 says you can't do what you want to do.  (it say "must be consistent rather than "MUST BE consistent", but RFC1813 is not a standards-track document).  In any case the write verifier handling was not designed to support what you want to do, even though client implementations might well work as you would wish.

  *   The server returns a write verifier upon successful completion of the
  *   COMMIT.  The write verifier is used by the client to determine if the
  *   server has restarted or rebooted between the initial WRITE(s) and the
  *   COMMIT.

Note that it says "the server"  implying there is just one.

  *   The client does this by comparing the write verifier
  *   returned from the initial writes and the verifier returned by the
  *   COMMIT operation.

"The write verifier" (singular) "returned from the initial writes" only make sense if the restriction
specified in RFC1813 is adhered to here as well.

  *   So, the client would compare COMMIT verifier with Previous WRITE verifier.

Except that that are multiple previous WRITE verifiers.

  *   And that allows SERVER to send "different" VERIFIER on different WRITE Operations

It can send them, but it is not clear how clients are supposed to deal with them since the assumption (in both specs) is that a single server will return the same one on successive writes.

From: Mehta, Viral <[email-masked]>
Sent: Thursday, June 21, 2018 4:50 AM
To: [email-masked]; Pawlowski, Brian <[email-masked]>; [email-masked]; Noveck, David <[email-masked]>; [email-masked]
Subject: RE: RFC 1813 - NFS questions on write verifier

RFC 7530 is bit more clear,

The server returns a write verifier upon successful completion of the
   COMMIT.  The write verifier is used by the client to determine if the
   server has restarted or rebooted between the initial WRITE(s) and the
   COMMIT.  The client does this by comparing the write verifier
   returned from the initial writes and the verifier returned by the
   COMMIT operation.

So, the client would compare COMMIT verifier with Previous WRITE verifier.
And that allows SERVER to send "different" VERIFIER on different WRITE Operations.

From: Mehta, Viral
Sent: Tuesday, June 12, 2018 10:32 AM
To: [email-masked]'; [email-masked]'; [email-masked]'
Cc: [email-masked]'; [email-masked]'
Subject: RFC 1813 - NFS questions on write verifier

Hi,

I had a question on NFSv3 server behaviour for unstable WRITEs in clustered environment.

As we know,
NFS client sends WRITE op with stable or unstable flag. If the client sends WRITE op with UNSTABLE flag, and NFS server responses
with the same flag, it also sends VERIFIER cookie. Later, when NFS server receives COMMIT operation, it sends the same VERIFIER cookie back and so
the NFS client is happy.

If in between WRITE and COMMIT operations, NFS server was crashed and rebooted, it sends the new VERIFIER (most of the implementation choose to send boot time of the NFS server)
and so the client should perform WRITE again and it is again happy.

It all works if there is single node where NFS server is running. What if NFS server is clusterized / running in scale-out environment ? I wanted to understand how is this VERIFIER maintained ?

For e.g., if one node "N1" in scale out receives NFS WRITE request and the UNSTABLE WRITE actually happened on other node "N2".
i.e., Node "N1" forwarded WRITE request to node "N2" since data was on the node "N2".
Now, if node "N2" crashes before it receives COMMIT, which VERIFIER is sent back to the client ? RFC-1813 says that

"This cookie must be
         consistent during a single boot session and must be
         unique between instances of the NFS version 3 protocol
         server where uncommitted data may be lost.
"

Since there may not be single NFS server instance, and if NFS server process is running on all scale out nodes. What is the verifier cookie that is sent back ?
And when is that verifier cookie updated ?

PS - It looks like real NFS client implementations are tracking COOKIE on PER WRITE basis and not PER NFS Server instance bases as mentioned in RFC.
         But, while implementing NFS server, is that fair enough to assume above scenario ?

Thanks,
Viral

---

From: Rick Macklem <[email-masked]>
Date: Thu, 21 Jun 2018 12:24:12 +0000

Just a data point...The FreeBSD client keeps the write verifier on
a per-mount basis and assumes that it will be the same for all Writes
unless the server has rebooted and requires the Writes to be redone.

rick

________________________________________
From: nfsv4 <[email-masked]> on behalf of Noveck, David <[email-masked]>
Sent: Thursday, June 21, 2018 7:37:47 AM
To: Mehta, Viral; [email-masked]; Pawlowski, Brian; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] RFC 1813 - NFS questions on write verifier

First of all, I’ve cc’d the v4 working group, which is a good idea in general when you ask questions about these specs, especially since most of the email addresses on your to-list are probably no longer active (e.g. anything ending eng.sun.com).

I think it is less clear.  RFC1813 says you can’t do what you want to do.  (it say “must be consistent rather than “MUST BE consistent”, but RFC1813 is not a standards-track document).  In any case the write verifier handling was not designed to support what you want to do, even though client implementations might well work as you would wish.

  *   The server returns a write verifier upon successful completion of the
  *   COMMIT..  The write verifier is used by the client to determine if the
  *   server has restarted or rebooted between the initial WRITE(s) and the
  *   COMMIT..

Note that it says “the server”  implying there is just one.

“The write verifier” (singular) “returned from the initial writes” only make sense if the restriction
specified in RFC1813 is adhered to here as well.

  *   And that allows SERVER to send “different” VERIFIER on different WRITE Operations

So, the client would compare COMMIT verifier with Previous WRITE verifier.
And that allows SERVER to send “different” VERIFIER on different WRITE Operations.

From: Mehta, Viral

Hi,

As we know,

For e.g., if one node “N1” in scale out receives NFS WRITE request and the UNSTABLE WRITE actually happened on other node “N2”.
i.e., Node “N1” forwarded WRITE request to node “N2” since data was on the node “N2”.
Now, if node “N2” crashes before it receives COMMIT, which VERIFIER is sent back to the client ? RFC-1813 says that

“This cookie must be
“

PS – It looks like real NFS client implementations are tracking COOKIE on PER WRITE basis and not PER NFS Server instance bases as mentioned in RFC.
         But, while implementing NFS server, is that fair enough to assume above scenario ?

Thanks,
Viral

---

From: "Noveck, David" <[email-masked]>
Date: Thu, 21 Jun 2018 14:30:08 +0000

When you speak of the server being rebooted, are you only keying off a change of verifier, or can you find out about a reboot (for these purposes) by seeing a change in the generation reported in the client_owner structure.   Although I haven't looked at this issue in detail, my impression had been that these two senses in which the client might have been rebooted were kept separate.   If others have read the spec differently (and particularly if they have written code based on that interpretation), there might be a need to clarify this somewhere.

-----Original Message-----
From: Rick Macklem <[email-masked]> 
Sent: Thursday, June 21, 2018 8:24 AM
To: Noveck, David <[email-masked]>; Mehta, Viral <[email-masked]>; [email-masked]; Pawlowski, Brian <[email-masked]>; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: RFC 1813 - NFS questions on write verifier

Just a data point...The FreeBSD client keeps the write verifier on a per-mount basis and assumes that it will be the same for all Writes unless the server has rebooted and requires the Writes to be redone.

rick

"The write verifier" (singular) "returned from the initial writes" only make sense if the restriction specified in RFC1813 is adhered to here as well.

From: Mehta, Viral

Hi,

As we know,
NFS client sends WRITE op with stable or unstable flag. If the client sends WRITE op with UNSTABLE flag, and NFS server responses with the same flag, it also sends VERIFIER cookie. Later, when NFS server receives COMMIT operation, it sends the same VERIFIER cookie back and so the NFS client is happy.

If in between WRITE and COMMIT operations, NFS server was crashed and rebooted, it sends the new VERIFIER (most of the implementation choose to send boot time of the NFS server) and so the client should perform WRITE again and it is again happy.

"This cookie must be

Thanks,
Viral

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 21 Jun 2018 12:05:00 -0400

I’m not aware of anything in the current specs that state the client is allowed to assume that the writeverf is a global constant, which is why the Linux kernel tracks on a per-file basis. However it is clear from the semantics of COMMIT that the writeverf is required to be at least global to the file, since the client is required to match the cookies returned by each WRITE to those returned by other WRITEs and the final COMMIT.

That said, if there are clients such as FreeBSD out there that assume the verifier is global, then perhaps we should introduce errata to RFC5661 and RFC7530 that clarify that this is now a requirement upon server implementations.

> On Jun 21, 2018, at 08:24, Rick Macklem <[email-masked]> wrote:
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Mehta, Viral" <[email-masked]>
Date: Sun, 24 Jun 2018 10:15:46 +0000

I agree and Linux has definitely interpreted it smartly, here. 

To some extent, even making VERF cookie per mount basis is also just one more interpretation of 
" cookie must be  consistent during a single boot session" when the same NFS server can serve multiple exports.

I opened an errata against 7530 and 5661 RFCs to make it bit more clear for server implementations.

Thanks,
Viral

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, June 21, 2018 9:35 PM
To: Rick Macklem
Cc: Noveck, David; Mehta, Viral; [email-masked]; Pawlowski, Brian; [email-masked]; Michael R Eisler; IETF NFSv4 WG Mailing List
Subject: Re: [nfsv4] RFC 1813 - NFS questions on write verifier

> Just a data point...The FreeBSD client keeps the write verifier on a 
> per-mount basis and assumes that it will be the same for all Writes 
> To: Mehta, Viral; [email-masked]; Pawlowski, Brian; 
> [email-masked]; [email-masked]
> Cc: [email-masked]
> “The write verifier” (singular) “returned from the initial writes” 
> only make sense if the restriction specified in RFC1813 is adhered to here as well.
> 
> 
> To: [email-masked]; Pawlowski, Brian 
> <[email-masked]>; [email-masked]; Noveck, 
> David <[email-masked]>; [email-masked]
> NFS client sends WRITE op with stable or unstable flag. If the client 
> sends WRITE op with UNSTABLE flag, and NFS server responses with the 
> same flag, it also sends VERIFIER cookie. Later, when NFS server receives COMMIT operation, it sends the same VERIFIER cookie back and so the NFS client is happy.
> 
> If in between WRITE and COMMIT operations, NFS server was crashed and 
> rebooted, it sends the new VERIFIER (most of the implementation choose to send boot time of the NFS server) and so the client should perform WRITE again and it is again happy.
> 
> Now, if node “N2” crashes before it receives COMMIT, which VERIFIER is 
> sent back to the client ? RFC-1813 says that
> 
> 
> “This cookie must be