Subject: [nfsv4] pNFS - file commit

From: Marc Eshel <[email-masked]>
Date: Wed, 15 Mar 2006 22:35:37 -0800

We had a long thread on the issue of file commit and it was agreed that we 
need to keep the NFS recovery model for write/commit using the verifier 
and I agree. I just recently tested write/commit to data server and found 
some expected performance problems that I would like to address. The 
problem is that when you commit a data block on a DS and write it to disk 
it is useless if you don't also update and commit to disk some other 
metadata like the indirect blocks (pointers to data blocks) and some times 
the inode too. Yes, I know that I am describing a specific implementation 
of the server but I suspect that other servers/file systems might have 
similar structures. So if I write to 10 DSs one data block to each DS and 
commit them independently to each DS I have to do 10 IOs for the data 
blocks and 10 IOs to the shared indirect block. I should be able to do 
only one IO for the indirect block. I need a way to have a coordinated 
commit among all the machines that make the pNFS server.

Option 1 is to add an optional flag to file commit that when the flag is 
on the DS needs to write the data only to disk but not the metadata. If 
the flag is off it behave exactly like specified in rfc 3530. When layout 
commit is called the MDS will also commit the metadata. In other words 
with the flag off it is data and metadata commit and with it on just 
commit data.

Option 2 instruct the pNFS client to keep all data buffers until it 
completes a successful layout commit. The client will still do the commit 
to each DS but release the buffers only after the layout commit.

Option is better because it gives the client the flexibility to 
distinguish between the to main reasons that it calls commit. One is when 
there is memory pressure and it need to free some buffers quickly. It will 
do the commit with the flag off (like the old commit) and it is done. The 
second reason is when in gets a request to sync the file. In this case the 
client will do it in a more coordinated way, it will first call commit to 
all DSs that have dirty data with the new flag on, followed by a layout 
commit call. Only after the layout commit it can free all the data 
buffers. 

Marc.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 16 Mar 2006 08:07:50 -0500

I don't understand quite what you mean by "file commit". Are you
proposing a new flag to WRITE which the clients pass to the data
servers? This sounds highly problematic.

Can't the data servers use the stateid set up by the metadata
server to determine what type of metadata consistency model
they might use? This would then become a private server-to-server
issue, as it (apparently) should be.

Tom.

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 08:51:57 -0800

"Talpey, Thomas" <[email-masked]> wrote on 03/16/2006 05:07:50 

> servers? This sounds highly problematic.

I mean the COMMIT rpc call that goes to the data server.

> issue, as it (apparently) should be.

I need to know if the client is planing to will follow the COMMIT with 
LAYOUTCOMMIT and keep the buffers until the LAYOUTCOMMIT.

---

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 16 Mar 2006 12:54:53 -0500

>LAYOUTCOMMIT and keep the buffers until the LAYOUTCOMMIT.

Aha - ok now I get it. We have discussed cache hints before, this
certainly falls in the same territory.

One way to look at this is you want a "stable_how" on COMMIT, like
there is for WRITE. Then the client will write UNSTABLE, and commit
with DATA_SYNC. Currently the commit semantic is basically FILE_SYNC.

I note that COMMIT doesn't take a stateid (why not?), but because
of the layout being granted, the server can sort of go from the cfh
to it, since the layout guarantees exclusive access while it's outstanding.
So I am thinking your semantic can still be implemented without making
it explicit in the protocol.

I think it's a little weird for the client to be playing this game however.
It has to make the decision to keep a copy of the COMMITted data
until the layout is committed. Do we want all clients to do this, just in
case some servers benefit? In other words, do you expect all clients
to do this?

Tom.

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 10:07:36 -0800

"Talpey, Thomas" <[email-masked]> wrote on 03/16/2006 09:54:53 

> with DATA_SYNC. Currently the commit semantic is basically FILE_SYNC.
yes.

> case some servers benefit? In other words, do you expect all clients
 
This is way I want to make it explicit in the protocol, so the client can 
tell the server if it decided to keep the buffer until the layout commit 
or not. A client that will decide to ignore this optional flag will work 
fine with the exception of performance degradation with some servers.

Marc. 

> >> >We had a long thread on the issue of file commit and it was agreed 
that 
> >we 

> >> >problem is that when you commit a data block on a DS and write it to 

> >disk 

> >> >metadata like the indirect blocks (pointers to data blocks) and some 

> >times 
> >> >similar structures. So if I write to 10 DSs one data block to each 
DS 
> >and 
> >> >Option 1 is to add an optional flag to file commit that when the 
flag 
> >is 

> >> >distinguish between the to main reasons that it calls commit. One is 

> >when 
> >> >there is memory pressure and it need to free some buffers quickly. 
It 
> >will 
> >> >do the commit with the flag off (like the old commit) and it is 
done. 
> >The 
> >> >second reason is when in gets a request to sync the file. In this 
case 
> >the 
> >> >client will do it in a more coordinated way, it will first call 
commit 
> >to

---

From: Dean Hildebrand <[email-masked]>
Date: Thu, 16 Mar 2006 14:44:36 -0500

Marc,

Can you be a little more specific on how the protocol would change and 
how the pNFS file layout driver will use the changes in a generic way 
for all back-end file systems supporting the pNFS NFSv4 file layout type?

Dean

Marc Eshel wrote:

>>>"Talpey, Thomas" <[email-masked]> wrote on 03/16/2006 
>>>      
>>>
>05:07:50 
>  
>
>>>AM:
>>>
>>>      
>>>
>>to it, since the layout guarantees exclusive access while it's 
>>    
>>
>outstanding.
>  
>
>>I think it's a little weird for the client to be playing this game 
>>    
>>
>however.
>  
>
>>until the layout is committed. Do we want all clients to do this, just 
>>    
>>
>in
>  
>
>>>>>need to keep the NFS recovery model for write/commit using the 
>>>>>          
>>>>>
>verifier 
>  
>
>>>>>and I agree. I just recently tested write/commit to data server and 
>>>>>          
>>>>>
>>>found 
>>>      
>>>
>>>>>the inode too. Yes, I know that I am describing a specific 
>>>>>          
>>>>>
>>>implementation 
>>>      
>>>
>>>>>of the server but I suspect that other servers/file systems might 
>>>>>          
>>>>>
>have 
>  
>
>>>>>commit them independently to each DS I have to do 10 IOs for the 
>>>>>          
>>>>>
>data 
>  
>
>>>>>blocks and 10 IOs to the shared indirect block. I should be able to 
>>>>>          
>>>>>
>do 
>  
>
>>>>>only one IO for the indirect block. I need a way to have a 
>>>>>          
>>>>>
>coordinated 
>  
>
>>>>>on the DS needs to write the data only to disk but not the metadata. 
>>>>>          
>>>>>
>If 
>  
>
>>>>>the flag is off it behave exactly like specified in rfc 3530. When 
>>>>>          
>>>>>
>>>layout 
>>>      
>>>
>>>>>commit is called the MDS will also commit the metadata. In other 
>>>>>          
>>>>>
>words 
>  
>
>>>>>completes a successful layout commit. The client will still do the 
>>>>>          
>>>>>
>>>commit 
>>>      
>>>
>>>>>all DSs that have dirty data with the new flag on, followed by a 
>>>>>          
>>>>>
>layout 
>  
>

-- 
Dean Hildebrand
Ph.D. Candidate
University of Michigan

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 12:58:25 -0800

The protocol change is a new flag on the COMMIT rpc that indicates to the 
server if this is data commit only or data and metadata commit. The 
default when the flag is off is the old COMMIT rpc behavior of commit data 
and metadata. How it will impact the Linux file layout driver is an 
implementation issue that we can have a more detailed discussion. The 
general idea is that when the driver is called to sync a file it will send 
COMMIT calls to all DSs that are referenced by the page list to be 
committed, send a layout commit to the MDS, and only than mark the pages 
clean. If the driver is called just to commit few pages because of memory 
pressure it will use the current logic (not using the new flag).
I though that you requested something similar on the last thread that we 
had on this issue. you said "maybe we need a new NFSv4 operation to sync 
data to disk on the data servers?  Something with a name like 
DISTRIBUTED_COMMIT or DCOMMIT."

Marc.

---

From: Garth Goodson <[email-masked]>
Date: Thu, 16 Mar 2006 14:02:23 -0800

Marc Eshel wrote:
> clean.

This is something that concerns me.  We wanted to avoid having to send 
LAYOUTCOMMITs so frequently, as well, we wanted to avoid having to keep 
the data around for the LAYOUTCOMMIT.  LAYOUTCOMMIT in some 
implementations may be fairly heavy weight (e.g., extending the file 
size on each data server).  So to some extent, depending on 
implementation, you are trading off IOs during COMMIT for I/Os during 
LAYOUTCOMMIT.

One last thing.  We would have to check whether it is in the scope of 
v4.1 to change existing operations (i.e., add a flag to COMMIT).  My 
impression was that existing operations can not be modified by a minor 
extension.

-Garth
-Garth

  If the driver is called just to commit few pages because of memory
>>for all back-end file systems supporting the pNFS NFSv4 file layout 
> 
> type?
> 
>>Dean
>>
> 
> 
>

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 14:18:14 -0800

> implementation, you are trading off IOs during COMMIT for I/Os during 

I am not sure if you are saying that you have extra IOs or the IOs just 
done at a different time. The problem that I am trying to solve will 
reduces the number of IOs. This will be done only when you know on the 
client side that the operation will have to add with a LAYOUTCOMMIT so 
there is no extra layout commit calls just delaying the release of the 
buffers to after the call. When you sync the file you know that it will 
end with a layout commit.

> impression was that existing operations can not be modified by a minor 

It can be done as a new rpc (PNFS_COMMIT) if we can not add a flag to the 
existing COMMIT rpc.

---

From: Garth Goodson <[email-masked]>
Date: Thu, 16 Mar 2006 14:52:06 -0800

Marc Eshel wrote:
>>>The protocol change is a new flag on the COMMIT rpc that indicates to 
> 
> the 
> 
>>>default when the flag is off is the old COMMIT rpc behavior of commit 
> 
> data 
> 
>>>general idea is that when the driver is called to sync a file it will 
> 
> send 
> 
>>>committed, send a layout commit to the MDS, and only than mark the 
> 
> pages 
> 
>>>clean.
>>

I'm saying the more often you call LAYOUTCOMMIT, the more work some 
implementations have to do.  For example, imagine you are writing 
sequentially to a new file.  If you do a LAYOUTCOMMIT for every 5 writes 
vs. every 50 writes you must sync the attributes on the data servers 10x 
more.  This requires much more work and many more I/Os.  You won't have 
this problem in your implementation (since you are using a cluster file 
system backend), but other implementations may.  As well, it would be 
better performance if you could ammortize the LAYOUTCOMMITs over many, 
many writes.  For example, a single client writing a multi-gigabit file 
need only really do 1 LAYOUTCOMMIT, but you would be requiring it to do 
  many more LAYOUTCOMMITs.

I'm just trying to point out that your solution will impact certian 
implementations and help out others.  I think the solution requires a 
bit more consideration of the overall impact (for all implementations 
e.g., those that don't share metadata across data servers -- these 
actually do require the full commit at each data server).

-Garth

>>>I though that you requested something similar on the last thread that 
> 
> we 
> 
>>>had on this issue. you said "maybe we need a new NFSv4 operation to 
> 
> sync 
> 
>>>Dean Hildebrand <[email-masked]> wrote on 03/16/2006 11:44:36 
> 
> AM:
> 
>>>
>>>>Marc,
>>>>

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 15:01:25 -0800

> need only really do 1 LAYOUTCOMMIT, but you would be requiring it to do 

Like I said in the last note the client will not do any extra layout 
commit calls. Only operations like sync file which you know that they need 
to end with layout commit with this new commit option.
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Thu, 16 Mar 2006 15:28:59 -0800

Marc Eshel wrote:
>>>Garth Goodson <[email-masked]> wrote on 03/16/2006 02:02:23 
> 
> PM:
> 
>>>
>>>>Marc Eshel wrote:
>>>>
>>>>
>>>>LAYOUTCOMMITs so frequently, as well, we wanted to avoid having to 
> 
> keep 
> 
>>>I am not sure if you are saying that you have extra IOs or the IOs 
> 
> just 
> 
>>>buffers to after the call. When you sync the file you know that it 
> 
> will 
> 

You've introduced the semantic that clients MUST issue LAYOUTCOMMIT 
before flushing data from their caches.  For apps that do direct io I 
would imagine this would have an impact.  I believe you would be forcing 
them to do LAYOUTCOMMITs much more frequently.

-Garth

>>>It can be done as a new rpc (PNFS_COMMIT) if we can not add a flag to 
> 
> the 
> 
>>>existing COMMIT rpc.
>>>
>>>
>>>
>>>>>Dean Hildebrand <[email-masked]> wrote on 03/16/2006 
> 
> 11:44:36 
> 
>>>AM:
>>>
>>>
>>>>>>Marc,
>>>>>>
>>>>>>Can you be a little more specific on how the protocol would change 
> 
> and 
> 
>>>
>>>>>>how the pNFS file layout driver will use the changes in a generic 
> 
> way 
>

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 15:38:12 -0800

> them to do LAYOUTCOMMITs much more frequently.

They need to issue LAYOUTCOMMIT only if they used the new commit call with 
the flag on, and they should use this option ONLY when they know that they 
will have to call LAYOUTCOMMIT anyhow. Maybe in the case of direct IO it 
should not use this new commit option so they don't have to call 
LAYOUTCOMMIT.
Marc.

---

From: Garth Goodson <[email-masked]>
Date: Thu, 16 Mar 2006 18:21:07 -0800

> commit call. Only after the layout commit it can free all the data 

I've been thinking about it and I'm not sure this should really be a 
problem when using a shared storage cluster filesystem (and I think it 
has been discussed before).

Imagine the following scenerio:

1) client does 10 WRITEs to 10 DSs

2) client starts doing COMMITs
    - on first COMMIT the data/inode/direct/indirect blocks should only 
get written out if they are dirty.  Let's assume due to the writes that 
they are.  The writeout of this dirty metadata/data will most likely 
trigger the writeout of metadata/data associated with the the writes 
that went to the other nodes.

3) client does rest of the COMMITs
   - implementation should be smart enough to check which blocks are 
dirty.  I would imagine that most of the dirty data/metadata has already 
been written out or is in the process of being written out and should 
not be scheduled again.

Basically, the implementation should be able to track what blocks are 
dirty and only sync them.  NFS clients often issue COMMITs for the whole 
file, or even if the commits are issued for parts of the file, the 
server can be smart enough to sync enough data/metadata on the first 
COMMIT as to avoid multiple I/Os due to multiple COMMITs.

In this scenario you still have the overhead of extra COMMIT RPCs but 
not of the extra I/Os.  As well, you may be able to avoid the overhead 
of the extra 10 data I/Os (depending on where they are in the stripe, etc).

-Garth

---

From: "Talpey, Thomas" <[email-masked]>
Date: Thu, 16 Mar 2006 21:20:21 -0500

>impression was that existing operations can not be modified by a minor 

This is easy to answer - we would need a new operation. Changing
the arguments and results of an existing operation is illegal. In addition
to being a really lousy idea!

Again, my main concern with the proposal is that it seems to require
clients to implement a very particular COMMIT method which retains
data beyond the COMMIT operation, in case the metadata LAYOUTCOMMIT
later failed. The complexity of this on the multitude of clients, to me,
negates any incremental benefit at the rather limited class of server
which might benefit. But, I would be interested to see a full proposal.

Tom.

---

From: Marc Eshel <[email-masked]>
Date: Thu, 16 Mar 2006 20:48:15 -0800

> > We had a long thread on the issue of file commit and it was agreed 
that we 
verifier 
found 
disk 
times 
implementation 

If 
layout 

commit 
when 
layout 
> Imagine the following scenerio:

I don't have to imagine I am looking at traces :)

> that went to the other nodes.
You assume that the metadata (indirect block pointers) are in the shared 
block after the write, but this not the case the metadata is shipped to 
the metadata node when the client commits the data and than you can tell 
the metadata node to commit the metadata or not.

> been written out or is in the process of being written out and should 

I guess we are no smart :(

> of the extra 10 data I/Os (depending on where they are in the stripe, 
etc).

I really don't want get in to discussion of our server internals. I don't 
think the rest of the people on the mailing list care how it works or how 
it should work. If there is no one else out there that can benefit from 
this proposal I will drop it. I though that PVFS2 can use it too.

Marc.

---

From: Dean Hildebrand <[email-masked]>
Date: Fri, 17 Mar 2006 10:59:54 -0500

>this proposal I will drop it. I though that PVFS2 can use it too.
The PVFS2 layout driver uses its own protocol for committing data.  
Having said that, I think there are cases with every exported file 
system where applications will want to use the file layout driver for 
security reasons, etc, so it is an issue for most file systems that 
wants to use pNFS.

Let me try to summarize the discussion so far (Let me know if I'm wrong):
"Data servers sometimes only manage data, and therefore would like a 
commit operation that only commits data to stable storage.  Metadata 
would be committed to disk by the MDS on layoutcommit."

Issues with allowing this way of syncing a file:
1) Will all data buffers need to be kept by the client until 
layoutcommit?  I think this was another option and not part of the above 
scenario.
2) Certain vendors/file systems such as Netapp want to commit data + 
metadata on each NFSv4 COMMIT operation to a data server.  How do we 
distinguish what commit operation to use for each file system?  File 
system attribute?
3) Does this scenario increase the number of required layoutcommits?  I 
don't think so.

Are the statement and the 3 issues correct?

-- 
Dean Hildebrand
Ph.D. Candidate
University of Michigan

---

From: Garth Goodson <[email-masked]>
Date: Fri, 17 Mar 2006 09:33:54 -0800

Dean Hildebrand wrote:
> 
>> I really don't want get in to discussion of our server internals. I 
>> don't think the rest of the people on the mailing list care how it 
>> works or how it should work. If there is no one else out there that 
>> can benefit from this proposal I will drop it. I though that PVFS2 can 
>> use it too.
>>
>> Marc.
>>  
>>

First, Marc, I'm sorry if I'm coming across as being overly critical. 
I'm just trying to understand the scenario in which this is occuring 
(i.e., what types of implementations lead to this problem; and whether 
this is a general pNFS file-layout problem or not).  Your proposal leads 
to a fairly major change in pNFS semantics (regarding LAYOUTCOMMIT).

I have two main concerns with your proposal:  1) complexity of the 
client -- figuring out what to do when; 2) mandating that data buffers 
are kept until LAYOUTCOMMIT.

> layoutcommit?  I think this was another option and not part of the above 

I think Marc is mandating that this is the case if the metadata has not 
been synced on COMMIT.  For example, a write that extends the file would 
have its data blocks written out, but not have any metadata written out 
describing the location of those data blocks.  In this case Marc would 
like to be able to sync that metadata in a separate operation after the 
data has been synced.

> distinguish what commit operation to use for each file system?  File 

I'm always hesistant to add too much to a protocol that is dependent 
upon implementation details as this tends to increase complexity of both 
the protocol and the implementations (especially in the client).  I 
think we need to decide whether we have two classes of implementations 
that truely require separate protocol support.

> 3) Does this scenario increase the number of required layoutcommits?  I 

In my view it can.  Previously only a single LAYOUTCOMMIT was necessary 
at the end of writing an entire file.  Now a client must have issue a 
LAYOUTCOMMIT before it can release any data buffers from the client's 
cache (even though that data has been synced).

-Garth

---

From: Dean Hildebrand <[email-masked]>
Date: Fri, 17 Mar 2006 12:39:47 -0500

Marc, can you please clarify this whole 'keeping the data buffer around' 
issue. 

Everyone seems to be stuck on it, but from what I remember you 
originally had 2 proposals, and only 1 involved keeping data buffers 
around.  We are talking about the other proposal, which does not require 
us to keep data buffers around.  Right?  Why do we need to keep data 
buffers around if the data is sync'd to disk?

Dean

Garth Goodson wrote:

>>> can benefit from this proposal I will drop it. I though that PVFS2 
>>> can use it too.
>>>
>>> Marc.
>>>  
>>>
>
> this is a general pNFS file-layout problem or not).  Your proposal 
> leads to a fairly major change in pNFS semantics (regarding 
> LAYOUTCOMMIT).
>
>> layoutcommit?  I think this was another option and not part of the 
>> above scenario.
>
>
> I think Marc is mandating that this is the case if the metadata has 
> not been synced on COMMIT.  For example, a write that extends the file 
> would have its data blocks written out, but not have any metadata 
> written out describing the location of those data blocks.  In this 
> case Marc would like to be able to sync that metadata in a separate 
> operation after the data has been synced.
>
> upon implementation details as this tends to increase complexity of 
> both the protocol and the implementations (especially in the client).  
> I think we need to decide whether we have two classes of 
> implementations that truely require separate protocol support.
>
>> 3) Does this scenario increase the number of required layoutcommits?  
>> I don't think so.
>>
>
> In my view it can.  Previously only a single LAYOUTCOMMIT was 
> necessary at the end of writing an entire file.  Now a client must 
> have issue a LAYOUTCOMMIT before it can release any data buffers from 
> the client's cache (even though that data has been synced).
>
> -Garth
>

-- 
Dean Hildebrand
Ph.D. Candidate
University of Michigan

---

From: Marc Eshel <[email-masked]>
Date: Fri, 17 Mar 2006 10:52:02 -0800

The second option was if we did not want to make any protocol changes we 
can just mandate that the client always does follow data commit with 
layout commit. So this one would increase the number of layout call which 
it sound like it will be bad for some backbends so would not consider this 
option anymore.

The first option was to change the protocol so the client can tell the 
data server that it will follow the data commit with a layout commit so 
the server don't have to do metadata commit until it get the layout commit 
and than do it once for data servers. This will NOT increase the number of 
layout commit calls since the client will use this option ONLY when it 
plans to issue the layout commit at the end of data commit. The client 
knows when it is processing a sync file operation that it will end with 
layout commit.
Marc.

---

From: Benny Halevy <[email-masked]>
Date: Sat, 18 Mar 2006 10:35:50 +0200

> it sound like it will be bad for some backbends so would not consider this 

Can we make this layout-type specific?

---

From: Marc Eshel <[email-masked]>
Date: Sat, 18 Mar 2006 09:51:12 -0800

> Can we make this layout-type specific?
Do you think that the above second option is useful for some layout-type? 
I was ready to drop it and consider only the first option (protocol 
change) if anything at all.
Marc.

---

From: Dean Hildebrand <[email-masked]>
Date: Mon, 20 Mar 2006 10:49:54 -0500

Benny Halevy wrote:

> Marc Eshel wrote:
>
>> The second option was if we did not want to make any protocol changes 
>> we can just mandate that the client always does follow data commit 
>> with layout commit. So this one would increase the number of layout 
>> call which it sound like it will be bad for some backbends so would 
>> not consider this option anymore.
>
>

This whole conversation only applies to the file layout type, and would 
only require changes to the file layout specification.  Updating 
metadata at the data servers definitely seems wasteful in many cases and 
a new data commit operation defintely seems needed.  Other layout 
drivers can sync data as they see fit, knowing that a layoutcommit will 
be issued at some point afterwards(as described in the spec).
Dean

>
>>
>> The first option was to change the protocol so the client can tell 
>> the data server that it will follow the data commit with a layout 
>> commit so the server don't have to do metadata commit until it get 
>> the layout commit and than do it once for data servers. This will NOT 
>> increase the number of layout commit calls since the client will use 
>> this option ONLY when it plans to issue the layout commit at the end 
>> of data commit. The client knows when it is processing a sync file 
>> operation that it will end with layout commit.
>> Marc.
>>>>>> I really don't want get in to discussion of our server internals. 
>>>>>> I don't think the rest of the people on the mailing list care how 
>>>>>> it works or how it should work. If there is no one else out there 
>>>>>> that can benefit from this proposal I will drop it. I though that 
>>>>>> PVFS2 can use it too.
>>>>>>
>>>>>> Marc.
>>>>>>
>>>>>>
>>>>
>>>> First, Marc, I'm sorry if I'm coming across as being overly 
>>>> critical. I'm just trying to understand the scenario in which this 
>>>> is occuring (i.e., what types of implementations lead to this 
>>>> problem; and whether 
>>>
>>
>>
>>>>> security reasons, etc, so it is an issue for most file systems 
>>>>> that wants to use pNFS.
>>>>>
>>>>> "Data servers sometimes only manage data, and therefore would like 
>>>>> a commit operation that only commits data to stable storage.  
>>>>> Metadata would be committed to disk by the MDS on layoutcommit."
>>>>>
>>>>> 2) Certain vendors/file systems such as Netapp want to commit data 
>>>>> + metadata on each NFSv4 COMMIT operation to a data server.  How 
>>>>> do we distinguish what commit operation to use for each file 
>>>>> system?  File system attribute?
>>>>
>>>>
>>>>
>>>> I'm always hesistant to add too much to a protocol that is 
>>>> dependent upon implementation details as this tends to increase 
>>>> complexity of both the protocol and the implementations (especially 
>>>> in the client). I think we need to decide whether we have two 
>>>> classes of implementations that truely require separate protocol 
>>>> support.
>>>>
>>>>
>>>>> 3) Does this scenario increase the number of required 
>>>>> layoutcommits? I don't think so.
>>>>>
>>>>
>>>> have issue a LAYOUTCOMMIT before it can release any data buffers 
>>>> from the client's cache (even though that data has been synced).
>>>>
>>>> -Garth
>>>>
>>>>

-- 
Dean Hildebrand
Ph.D. Candidate
University of Michigan