Subject: Re: [nfsv4] pNFS - file commit

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 10:43:42 -0700

I dropped the cc to the mailing list on the last note but is included in 
this one. 

Garth Goodson <[email-masked]> wrote on 10/03/2005 10:17:09 AM:

> It is the inverse of what you are saying that troubles me.  If a write 
> is written to a DS, then is it visible?  You are saying not until a 
> LAYOUTCOMMIT.  And what about a synchronous write to a DS (one that does 

> not require a COMMIT), when is that visible?  Are you going to require 
> roundtrip communication on every synchronous DS write to update the 
> attributes?
> I'm worried about overly specifying what the control protocol has to do. 

>   Your statement is innocent enough, but I think it depends on what is 
> visible before that LAYOUTCOMMIT is done.

I am not sure about other devices but with the file protocol using a file 
system as the data server all writes are visible immediately from that DS 
for all synchronous or asynchronous write.
Marc. 

> 
> Marc Eshel wrote:
> > Let me try to make a different statement to achieve what I really 
want. 
> > LAYOUTCOMMIT if it is done for the entire layout will grantee that all 
the 
> > data and metadata will be committed to disk. I should be able to do a 
> > writes to different data servers (no commits) and than with one 
> > LAYOUTCOMMIT to the MDS have it all on disk. Right?
> > Marc. 
> > 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 10:51:28 -0700

> for all synchronous or asynchronous write.

If you assume this, then why the resistance to having clients do COMMITs 
at the DSs (for async. writes)?  LAYOUTCOMMIT at the MDS will make the 
attributes visible.  In implementations where there is no difference 
between async and sync writes (e.g., if one is using a NVRAM backed 
write log), the COMMIT to the DS will be ignored.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 11:02:38 -0700

> write log), the COMMIT to the DS will be ignored.
I did not say that there is no difference between sync and async writes. 
What I said is that they are both visible immediately so there is no need 
for the commit on the DS to make it visible. You still need the commit to 
save it on disk. All I am saying is that this commit will come from the 
MDS.
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Mon, 03 Oct 2005 11:05:42 -0700

Marc Eshel wrote:
>>>I dropped the cc to the mailing list on the last note but is included 
> 
> in 
> 
>>>this one. 
>>>
>>>Garth Goodson <[email-masked]> wrote on 10/03/2005 10:17:09 
> 
> AM:
> 
>>>
>>>>LAYOUTCOMMIT.  And what about a synchronous write to a DS (one that 
> 
> does 
> 
>>>
>>>>I'm worried about overly specifying what the control protocol has to 
> 
> do. 
> 
>>>
>>>I am not sure about other devices but with the file protocol using a 
> 
> file 
> 
>>>system as the data server all writes are visible immediately from that 
> 
> DS 
> 

I'm just looking for motivation as to why?  Are you trying to avoid the 
extra communication between the client and the DSs, or are you getting 
at something more?  Initially, I thought you were worried about a DS 
failure after the DS COMMIT and before the LAYOUTCOMMIT, however your 
approach has to deal with the same scenario if sync. writes are used 
(unless you dictate that the attributes are updated on every sync. write).

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 11:20:28 -0700

> (unless you dictate that the attributes are updated on every sync. 
write).
My motivation as I said in the first note is to save I/Os to disk. I am 
not against commit to data server it should be allowed and some times it 
is required before client can free memory of written data. All I want is 
an operation on the MDS that will coordinate and make sure that all dirty 
data on the data servers is committed to disk and than update the metadata 
on the MDS once. It works better for cluster file systems.
Marc.

---

From: "William A.(Andy) Adamson" <[email-masked]>
Date: Mon, 03 Oct 2005 15:48:53 -0400

[email-masked] said:
> My motivation as I said in the first note is to save I/Os to disk. I am  not
> against commit to data server it should be allowed and some times it  is
> required before client can free memory of written data. All I want is  an
> operation on the MDS that will coordinate and make sure that all dirty  data
> on the data servers is committed to disk and than update the metadata  on the
> MDS once. It works better for cluster file systems. Marc. 

why not use LAYOUTCOMMIT?

the cluster file system could pay attention to or ignore COMMITS to DSs, and 
use LAYOUTCOMMIT as the operation on the MDS that corrdinates and makes sure 
that all the dirty data on DS is committed to disk, and then update the 
metadata.

i don't see anything in the LAYOUTCOMMIT description that prohibits this use.

-->Andy

-->Andy

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 13:24:54 -0700

"William A.(Andy) Adamson" <[email-masked]> wrote on 10/03/2005 

> > My motivation as I said in the first note is to save I/Os to disk.I am 
 not
> i don't see anything in the LAYOUTCOMMIT description that prohibits this 
use.
There is nothing that prohibits it but I would like a statement to the 
client implementer that any uncommitted data by the time LAYOUTCOMMIT is 
issued will be committed at that point by the MDS.
Marc.

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 3 Oct 2005 18:45:15 -0400 (EDT)

> i don't see anything in the LAYOUTCOMMIT description that prohibits this use.

In the version of the ops draft that I have, page 17 states:
[Note: the data should be written (and committed) to the appropriate
storage devices before the LAYOUTCOMMIT occurs.]

I don't mind leaving this statement in the draft, but we definitely need
an entire section describing how this will be done for NFSv4 file based
layout drivers and for everyone else.

I know PVFS2 clients send 'commit-like' operations directly to the storage
nodes (data servers), and it sounds like the object people can utilize the
OSD protocol (although, as Jim said, the complexity of this is worrying).
How about the block-based people?  Is any block based implementations out
there expecting to utilize the NFSv4 COMMIT operation?

I'll bet this is pre-mature for the first go-around of the protocol, but
since the COMMIT verifier model is inadequate for pNFS, maybe we need a
new NFSv4 operation to sync data to disk on the data servers?  Something
with a name like DISTRIBUTED_COMMIT or DCOMMIT.  Providing at least one
way for a layout driver to sync data to disk through NFS would seem
useful. LAYOUTCOMMIT would seem not to provide adaquate fault tolerance
support.

Without some performance experiments, I'm also not convinced that sending
COMMIT to the data servers is more scalable or more efficient in every
case.

Dean