Subject: RE: [nfsv4] pNFS - file commit

From: "Halevy, Benny" <[email-masked]>
Date: Mon, 3 Oct 2005 01:42:21 -0400

Marc, my comments below...

> -----Original Message-----
> From: [email-masked] 
> [mailto:[email-masked]]On Behalf Of
> Marc Eshel
> Sent: Sunday, October 02, 2005 11:46 PM
> To: [email-masked]
> Subject: [nfsv4] pNFS - file commit
> 
> 
> What does regular file commit to the metadata server mean for 
> a file with 
> a layout? I would like it to mean the same as it means for a 
> file without 
> layouts. 

right, COMMIT should mean and do the same thing it does for a
file without layouts.

> So the client has the option of calling 
> layout-commit to all the 
> data server or have one call to the metadata server that will 
> accomplish 
> the same. 

Hmm, the way I read section 3.6 in draft-welch-pnfs-ops-04.txt
the MDS has to flush dirty data to the data servers when it
gets a COMMIT op, but it doesn't say it has to COMMIT it there
as well (or the equivalent in other storage types).
I'm attaching two email on that subject from august for
reference.  My original intent was that the MDS will indeed
send COMMIT to the data servers to make sure the data reached
stable storage there (or write with (FILE|DATA)_SYNC4) otherwise
I think it fails, as a system, to meet with nfsv4 requirements
for COMMIT, e.g. if the data servers write asynchronously to their
local storage and the system restarts dirtily after the data server
acknowledged the client-sent COMMIT and before the data was written
to stable storage on the data servers.

> The benefit of the one call to the MDS is that the 
> metadata will 
> not be written multiple times to disk. The MDS must write 
> metadata to disk 
> for each DS layout-commit but when it is driven from the MDS 
> with file 
> commit it can request all DS to commit their data to disk and 
> when they 
> are all done make one update to the metadata on disk.
> Marc. 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

Subject: RE: [nfsv4] New pNFS ID available
From: "Noveck, Dave" <[email-masked]>
To: "Halevy, Benny" <[email-masked]>, Craig Everhart <[email-masked]>
Cc: [email-masked], [email-masked]
Date: Tue, 2 Aug 2005 14:16:14 -0400

Benny Halevy:
> One (maybe obvious) thing that needs to be said is that the 
> client can expect to read back the same data it wrote 
> through the metadata server but cannot assume anything about 
> the written data if reading from the data servers using the 
> layout.

> I think it makes sense to require from nfsv4 servers which 
> support pnfs that COMMIT (and synchronous writes) will 
> commit dirty data to stable storage on the data servers so 
> that it could be read via layouts.

I agree.

As to "cannot assume anything" with regard to asynchronous writes,
I'm wondering if that might be too liberal.  For example, if I
write 4K, an implementation might propagate all the odd bytes
immediately and the even bytes at COMMIT time.  Even though 
nobody would be that wierd, I'm wondering whether some restrictions
might make some applications easier.  For example, the blocks
protocol might reasonably require that all of the write that is
within one block happen at one time, so that if the write is on
a block boundary, you can't read a partially updated block.  

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]]
Sent: Tuesday, August 02, 2005 8:48 AM
To: Craig Everhart
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] New pNFS ID available

Craig Everhart wrote:
> 
> [email-masked] wrote on 08/01/2005 05:45:04 PM:
> 
>  > If client A holds a write layout on a region in a file, and client B
>  > does a READ to the metadata server that contains that region of that
>  > file, the server must ("MUST") recall the layout from client A.
>  >
>  > If client A does the read, I would expect the server to return
>  > bits that are not more out-of-date than the most recent LAYOUTCOMMIT
>  > from client A for the layout. If client A wants that read to return
>  > up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
>  > A layout is identified by {ClientID,FileHandle,offset,length,
>  > layout_type,iomode
>  > }*,
>  > so the server should have enough information to figure out if the
>  > requestor holds an outstanding layout.
> 
> Right.  Good.  Now how about the case I cited, where the client does a
> write to the meta server.  Should it assume that its layout is useless,
> since it can't tell when it could read its data back from the
> storage?  Clearly the NFS meta server can implement persistent
> storage any way it intends, and the only consistency on the data
> is that it shouldn't hand out further layouts to a device or a
> set of devices until its pending data could be read by future
> fetches from that device or those devices.

One (maybe obvious) thing that needs to be said is that the 
client can expect to read back the same data it wrote 
through the metadata server but cannot assume anything about 
the written data if reading from the data servers using the 
layout.

I think it makes sense to require from nfsv4 servers which 
support pnfs that COMMIT (and synchronous writes) will 
commit dirty data to stable storage on the data servers so 
that it could be read via layouts.

Benny

> 
> 
>  > In the case where client A holds a read-only layout, and client B
>  > does a read, there is no conflict.
>  >
>  > In the case where the reading client hasn't set a client ID, the
>  > server should (IMO) assume that any/all outstanding write layouts
>  > are held by clients other than the reading client, and recall them
>  > all before servicing the read.
>  >
>  > -jim
>  >
>  > * Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but I'm
>  > assuming that it is included for the purposes of this e-mail.
>  >
>  > --
>  > Jim Zelenka
>  > Software Engineer, Panasas, Inc.
>  > Accelerating Time to Results(TM) with Clustered Storage
>  > www.panasas.com
>  > 412.323-3500
> 
>                 Craig
> 
> Craig Everhart
> +1 919 543 2169 (tie 441 2169)
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

Subject: RE: [nfsv4] New pNFS ID available
From: "Noveck, Dave" <[email-masked]>
To: "Halevy, Benny" <[email-masked]>
Cc: Craig Everhart <[email-masked]>, "Zelenka, Jim" <[email-masked]>, [email-masked]
Date: Wed, 3 Aug 2005 08:34:25 -0400

> > Most certainly not.  The layout is a promise by the server that data 
> > can be read using it, and if that promise ceases to ever be valid, then
> > it will be recalled.  Once the WRITE returns, if the layout has not 
> > been recalled, you can use it.  If anything, including the WRITE,
> > renders it useless, it is the responsibility of the server to recall
> > it. 

> I agree, just the guarantee is on the mapping not on the 
> contents...  I'm not sure that your definition of "useless" 
> and Craig's are the same.  

Mine is "of no use" :-)

> Reading possibly stale data using 
> the layout may not be considered useless for many 
> applications that use NFS today.  At the same time many such 
> applications may use fsync() and close() to flush the client 
> and server buffers onto stable storage and expect basic 
> level of distributed data consistency so that after client A 
> writes data and syncs it using fsync()/close() the data will 
> be visible (persistently) to any other client.  

And typically until you do the fsync()/close() some of your
written data is in the buffer cache and others may not see it.
That means they may read data that could be described as 
"stale" in a certain sense, but I definitely think it would
be going way overboard to say that the fact that I had a
write which was on the wire rendered the file handles that
others had "useless".  They are still quite useful and if
I want to make them useful to get the precisely up-to-data
data, then all I have to do is WRITE it using a synchronous
or WRITE is using an async write and then do a COMMIT.

Note here that if I do an async WRITE and do not (yet) do a
COMMIT, all other clients are not really able to rely on 
getting up-to-date data, not only those getting the data
directly from the data server.  Normally, a READ will get the
updated data but there is no assurance of that.  If the 
server goes down in the interim, a client may read the old
data.

The basic point here is there is always going to be a period
of indeterminateness with regard to newly written data.  During
this period a read may get the old data or the new data or 
some mixture.  If the write is synchronous the period of
indeterminateness begins when the WRITE is put on the wire
and ends when it executed.  For async writes, it extends all
the way until the COMMIT is executed.  This has always been
true of NFS and pNFS follows along.  The period of indeterminateness
is not any longer.  It is only that you are far more likely
to see the old data when reading using the layout.  Nevertheless,
it is possible to see the old data when doing ordinary READ's
as well.  If you think this makes the READs or the data structures 
used to do the reads (filehandles, layouts) useless, then so
be it, but I would  not describe things that way.  I think it means 
that the writing client has decided, correctly or incorrectly,
that the application can function without concurrent readers 
seeing the new data instantaneously.

> Hence, my 
> proposal to require pnfs enabled servers to sync dirty data 
> to stable storage on the data servers on COMMIT and 
> synchronous writes.

I agree with this.  When you write to a meta-data server 
asynchronously, the server may propagate the written data
to the data servers (so that they are read by those using
layouts) but is not obliged to do so until the the COMMIT
occurs.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]]
Sent: Tuesday, August 02, 2005 9:01 AM
To: Noveck, Dave
Cc: Craig Everhart; [email-masked]; [email-masked]
Subject: Re: [nfsv4] New pNFS ID available

Noveck, Dave wrote:
>>>If client A holds a write layout on a region in a file, and client B
>>>does a READ to the metadata server that contains that region of that
>>>file, the server must ("MUST") recall the layout from client A.
>>>
>>>If client A does the read, I would expect the server to return
>>>bits that are not more out-of-date than the most recent LAYOUTCOMMIT
>>>from client A for the layout. If client A wants that read to return
>>>up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
>>>A layout is identified by {ClientID,FileHandle,offset,length,
>>>layout_type,iomode
>>>}*,
>>>so the server should have enough information to figure out if the
>>>requestor holds an outstanding layout. 
> 
> 
>>Right.  Good.  
> 
> 
> Not as far as I'm concerned.
> 
> 
>>Now how about the case I cited, where the client does a 
>>write to the meta server.  Should it assume that its layout is useless, 
>>since it can't tell when it could read its data back from the 
>>storage?  
> 
> 
> Most certainly not.  The layout is a promise by the server that data 
> can be read using it, and if that promise ceases to ever be valid, then
> it will be recalled.  Once the WRITE returns, if the layout has not 
> been recalled, you can use it.  If anything, including the WRITE,
> renders it useless, it is the responsibility of the server to recall
> it. 

I agree, just the guarantee is on the mapping not on the 
contents...  I'm not sure that your definition of "useless" 
and Craig's are the same.  Reading possibly stale data using 
the layout may not be considered useless for many 
applications that use NFS today.  At the same time many such 
applications may use fsync() and close() to flush the client 
and server buffers onto stable storage and expect basic 
level of distributed data consistency so that after client A 
writes data and syncs it using fsync()/close() the data will 
be visible (persistently) to any other client.  Hence, my 
proposal to require pnfs enabled servers to sync dirty data 
to stable storage on the data servers on COMMIT and 
synchronous writes.

> 
> 
>>Clearly the NFS meta server can implement persistent 
>>storage any way it intends, 
> 
> 
> OK so far.
> 
> 
>>and the only consistency on the data 
>>is that it shouldn't hand out further layouts to a device or a 
>>set of devices until its pending data could be read by future 
>>fetches from that device or those devices. 
> 
> 
> Absolutely false as far as I'm concerned.  It also has a consistency
> constraint with regard to layouts already outstanding (in addition
> to "further layouts").  While the WRITE is pending reads to the
> data server can give you the old or the new data (or a mixture I
> guess -- no atomicity requirement).  After the WRITE completes, they
> have to give you the new data.  If the server implements its storage
> in any way such that it cannot obey those constraints then it must
> recall the layouts to prevent reads being done that cannot be handled
> correctly.
> 
> -----Original Message-----
> From: Craig Everhart [mailto:[email-masked]]
> Sent: Monday, August 01, 2005 6:37 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] New pNFS ID available
> 
> 
> 
> [email-masked] wrote on 08/01/2005 05:45:04 PM:
> 
> 
>>If client A holds a write layout on a region in a file, and client B
>>does a READ to the metadata server that contains that region of that
>>file, the server must ("MUST") recall the layout from client A.
>>
>>If client A does the read, I would expect the server to return
>>bits that are not more out-of-date than the most recent LAYOUTCOMMIT
>>from client A for the layout. If client A wants that read to return
>>up-to-date bits, then it should LAYOUTCOMMIT or LAYOUTRETURN.
>>A layout is identified by {ClientID,FileHandle,offset,length,
>>layout_type,iomode
>>}*,
>>so the server should have enough information to figure out if the
>>requestor holds an outstanding layout. 
> 
> 
> Right.  Good.  Now how about the case I cited, where the client does a 
> write to the meta server.  Should it assume that its layout is useless, 
> since it can't tell when it could read its data back from the 
> storage?  Clearly the NFS meta server can implement persistent 
> storage any way it intends, and the only consistency on the data 
> is that it shouldn't hand out further layouts to a device or a 
> set of devices until its pending data could be read by future 
> fetches from that device or those devices. 
> 
> 
> 
>>In the case where client A holds a read-only layout, and client B
>>does a read, there is no conflict.
>>
>>In the case where the reading client hasn't set a client ID, the
>>server should (IMO) assume that any/all outstanding write layouts
>>are held by clients other than the reading client, and recall them
>>all before servicing the read.
>>
>>-jim
>>
>>* Well, draft-welch-pnfs-ops-03 has iomode as an open issue, but I'm
>>assuming that it is included for the purposes of this e-mail.
>>
>>-- 
>>Jim Zelenka
>>Software Engineer, Panasas, Inc.
>>Accelerating Time to Results(TM) with Clustered Storage
>>www.panasas.com
>>412.323-3500 
> 
> 
>                 Craig
> 
> Craig Everhart
> +1 919 543 2169 (tie 441 2169)
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Marc Eshel <[email-masked]>
Date: Mon, 3 Oct 2005 09:30:57 -0700

Writing to the MDS will having layout outstanding is to confusing. I think 
that we should either use layouts for all read/writes that will direct all 
clients to go to the same DS for a given byte range or just do I/O through 
the MDS to satisfy the requirement that all data must be reachable through 
the MDS. I was suggesting that full file commit on the MDS will mean file 
commit on all data servers for data that was written directly to the DS so 
the metadata can be updated once.
Marc. 

[email-masked] wrote on 10/02/2005 10:42:21 PM:

> ----- Message from "Noveck, Dave" <[email-masked]> on Tue, 2
> Aug 2005 14:16:14 -0400 -----
> 
> To:
> 
> "Halevy, Benny" <[email-masked]>, Craig Everhart 
<[email-masked]>
> 
> cc:
> 
> [email-masked], [email-masked]
> 
> Subject:
> 
> RE: [nfsv4] New pNFS ID available
> 
> Benny Halevy:
> ----- Message from "Noveck, Dave" <[email-masked]> on Wed, 3
> Aug 2005 08:34:25 -0400 -----
> 
> To:
> 
> "Halevy, Benny" <[email-masked]>
> 
> cc:
> 
> Craig Everhart <[email-masked]>, "Zelenka, Jim" <jimz@panasas.
> com>, [email-masked]
> 
> Subject:
> 
> RE: [nfsv4] New pNFS ID available
>