Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 09:06:25 -0400

On Wed, 2010-07-07 at 15:05 +0300, Benny Halevy wrote:
> On Jul. 06, 2010, 23:40 +0300, Trond Myklebust <[email-masked]> wrote:
> > On Tue, 2010-07-06 at 15:20 -0400, [email-masked] wrote: 
> >> The COMMIT to the DS, ttbomk, commits data on the DS.  I see it as
> >> orthogonal to updating the metadata on the MDS (but perhaps I'm wrong).
> >> As sjoshi@bluearc mentioned, the LAYOUTCOMMIT provides a synchronization
> >> point, so even if the non-clustered server does not want to update
> >> metadata on every DS I/O, the LAYOUTCOMMIT could also be a trigger to
> >> execute whatever synchronization mechanism the implementer wishes to put
> >> in the control protocol.
> > 
> > As far as I'm aware, there are no exceptions in RFC5661 that would allow
> > pNFS servers to break the rule that any visible change to the data must
> > be atomically accompanied with a change attribute update.
> > 
> 
> Trond, I'm not sure how this rule you mentioned is specified.
> 
> See more in section 12.5.4 and 12.5.4.1. LAYOUTCOMMIT and change/time_modify
> in particular:
> 
>    For some layout protocols, the storage device is able to notify the
>    metadata server of the occurrence of an I/O; as a result, the change
>    and time_modify attributes may be updated at the metadata server.
>    For a metadata server that is capable of monitoring updates to the
>    change and time_modify attributes, LAYOUTCOMMIT processing is not
>    required to update the change attribute.  In this case, the metadata
>    server must ensure that no further update to the data has occurred
>    since the last update of the attributes; file-based protocols may
>    have enough information to make this determination or may update the
>    change attribute upon each file modification.  This also applies for
>    the time_modify attribute.  If the server implementation is able to
>    determine that the file has not been modified since the last
>    time_modify update, the server need not update time_modify at
>    LAYOUTCOMMIT.  At LAYOUTCOMMIT completion, the updated attributes
>    should be visible if that file was modified since the latest previous
>    LAYOUTCOMMIT or LAYOUTGET

I know. However the above paragraph does not state that the server
should make those changes visible to clients other than the one that is
writing.

Section 18.32.4 states that writes will cause the time_modified and
change attributes to be updated (if and only if the file data is
modified). Several other sections rely on this behaviour, including
section 10.3.1, section 11.7.2.2, and section 11.7.7.

The only 'special behaviour' that I see allowed for pNFS is in section
13.10, which states that clients can't expect to see changes
immediately, but that they must be able to expect close-to-open
semantics to work. Again, if this is to be the case, then the server
_must_ be able to deal with the case where client 1 dies before it can
issue the LAYOUTCOMMIT.

> > As I see it, if your server allows one client to read data that may have
> > been modified by another client that holds a WRITE layout for that range
> > then (since that is a visible data change) it should provide a change
> > attribute update irrespective of whether or not a LAYOUTCOMMIT has been
> > sent.
> 
> the requirement for the server in WRITE's implementation section 
> is quite weak: "It is assumed that the act of writing data to a file will
> cause the time_modified and change attributes of the file to be updated."
> 
> The difference here is that for pNFS the written data is not guaranteed
> to be visible until LAYOUTCOMMIT.  In a broader sense, assuming the clients
> are caching dirty data and use a write-behind cache, application-written data
> may be visible to other processes on the same host but not to others until
> fsync() or close() - open-to-close semantics are the only thing the client
> guarantees, right?  Issuing LAYOUTCOMMIT on fsync() and close() ensure the
> data is committed to stable storage and is visible to all other clients in
> the cluster.

See above. I'm not disputing your statement that 'the written data is
not guaranteed to be visible until LAYOUTCOMMIT'. I am disputing an
assumption that 'the written data may be visible without an accompanying
change attribute update'.

Trond

> Benny
> 
> > If your MDS is incapable of determining whether or not data has changed
> > on the DSes, then it should probably recall the WRITE layout if someone
> > tries to read data that may have been modified. Said server also needs a
> > strategy for determining if a data change occurred if the client that
> > held the WRITE layout died before it could send the LAYOUTCOMMIT.
> > 
> > Cheers
> >    Trond
> > 
> >> -Dan
> >>
> >>> -----Original Message-----
> >>> From: Andy Adamson [mailto:[email-masked]] 
> >>> Sent: Tuesday, July 06, 2010 6:38 AM
> >>> To: Muntz, Daniel
> >>> Cc: [email-masked]; [email-masked]; [email-masked]
> >>> Subject: Re: 4.1 client - LAYOUTCOMMIT & close
> >>>
> >>>
> >>> On Jul 2, 2010, at 5:46 PM, <[email-masked]> wrote:
> >>>
> >>>> By "extremely lame server" I assume you mean any pNFS server that
> >>>> doesn't have a cluster FS on the back end.
> >>>
> >>> No, I mean a pNFS file layout type server that depends upon 
> >>> the 'hint'  
> >>> of file size given by LAYOUTCOMMIT. This does not mean that the file  
> >>> system has to be a cluster FS.
> >>>
> >>> If COMMIT through MDS is set, the MDS to DS protocol (be it a 
> >>> cluster  
> >>> FS or not) ensures the data is "commited" on the DSs.  
> >>> LAYOUTCOMMIT is  
> >>> not needed.
> >>>
> >>> If COMMITs are sent to the DSs (or FILE_SYNC writes), then 
> >>> the MDS to  
> >>> DS protocol (be it a cluster FS or not) should kick off a 
> >>> back-end DS  
> >>> to MDS communication to update the file size on the MDS.
> >>>
> >>> What I consider an 'extremely lame pNFS file layout server' is one  
> >>> that requires COMMITs to the DS and then depends upon the 
> >>> LAYOUTCOMMIT  
> >>> to communicate the commited data size to the MDS.
> >>>
> >>> -->Andy
> >>>
> >>>
> >>>> So while this might work
> >>>> well for NetApp (as long as NetApp never ships a non-clustered  
> >>>> pNFS), it
> >>>> might break others, or at least severely impact their 
> >>> performance.   
> >>>> For
> >>>> example, will the Solaris pNFS server work correctly without
> >>>> LAYOUTCOMMIT?  IMHO, the client MUST issue the appropriate  
> >>>> LAYOUTCOMMIT,
> >>>> but the server is free to handle it as a no-op if the server
> >>>> implementation does not need to utilize the payload.
> >>>>
> >>>>  -Dan
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: [email-masked]
> >>>>> [mailto:[email-masked]] On Behalf Of Andy Adamson
> >>>>> Sent: Friday, July 02, 2010 8:41 AM
> >>>>> To: Sandeep Joshi
> >>>>> Cc: [email-masked]; [email-masked]
> >>>>> Subject: Re: 4.1 client - LAYOUTCOMMIT & close
> >>>>>
> >>>>>
> >>>>> On Jul 1, 2010, at 8:07 PM, Sandeep Joshi wrote:
> >>>>>
> >>>>> Hi Sandeep
> >>>>>
> >>>>>>
> >>>>>> In certain cases, I don't see layoutcommit on a file at all even
> >>>>>> after doing many writes.
> >>>>>
> >>>>> FYI:
> >>>>>
> >>>>> You should not be paying attention to layoutcommits  - they have no
> >>>>> value for the file layout type.
> >>>>>
> >>>>> From RFC 5661:
> >>>>>
> >>>>> "The LAYOUTCOMMIT operation commits chages in the layout 
> >>> represented
> >>>>> by the current filehandle, client ID (derived from the 
> >>> session ID in
> >>>>> the preceding SEQUENCE operation), byte-range, and stateid."
> >>>>>
> >>>>> For the block layout type, this sentence has meaning in that
> >>>>> there is
> >>>>> a layoutupdate4 payload that enumerates the blocks that 
> >>> have changed
> >>>>> state from being 'handed out' to being 'written'.
> >>>>>
> >>>>> The file layout type has no layoutupdate4 payload, and the
> >>>>> layout does
> >>>>> not change due to writes, and thus the LAYOUTCOMMIT call 
> >>> is useless.
> >>>>>
> >>>>> The only field in the LAYOUTCOMMIT4args that might possibly
> >>>>> be useful
> >>>>> is the loca_last_write_offset which tells the server what 
> >>> the client
> >>>>> thinks is the EOF of the file after WRITE. It is an extremely lame
> >>>>> server (file layout type server) that depends upon clients for this
> >>>>> info.
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Client side operations:
> >>>>>>
> >>>>>> open
> >>>>>> write(s)
> >>>>>> close
> >>>>>>
> >>>>>>
> >>>>>> On server side (observed operations):
> >>>>>>
> >>>>>> open
> >>>>>> layoutget's
> >>>>>> close
> >>>>>>
> >>>>>>
> >>>>>> But, I do not see laycommit at all. In terms data written
> >>>>> by client
> >>>>>> it is about 4-5MB.
> >>>>>>
> >>>>>> When does client issue laycommit?
> >>>>>
> >>>>> The latest linux client sends a layout commit when the VFS does a
> >>>>> super_operations.write_inode call which happens when the 
> >>> metadata of
> >>>>> an inode needs updating. We are seriously considering removing the
> >>>>> layoutcommit call from the file layout client.
> >>>>>
> >>>>> -->Andy
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> regards,
> >>>>>>
> >>>>>> Sandeep
> >>>>>>
> >>>>>> --
> >>>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-nfs"
> >>>>>> in
> >>>>>> the body of a message to [email-masked]
> >>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>>> --
> >>>>> To unsubscribe from this list: send the line "unsubscribe
> >>>>> linux-nfs" in
> >>>>> the body of a message to [email-masked]
> >>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>>>
> >>>>>
> >>>
> >>>
> >>>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> >> the body of a message to [email-masked]
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> > 
> > 
> > 
> >

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 09:18:16 -0400

> change attribute update'.

In other words, I'd expect the following scenario to give the same
results in NFSv4.1 w/pNFS as it does in NFSv4:

Client 1			Client 2
========			========

OPEN foo
READ
CLOSE
				OPEN
				LAYOUTGET ...
				WRITE via DS
				<dies>...
OPEN foo
verify change_attr
READ if above WRITE is visible
CLOSE

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 10:03:36 -0400

On Wed, 2010-07-07 at 16:51 +0300, Benny Halevy wrote:
> On Jul. 07, 2010, 16:18 +0300, Trond Myklebust <[email-masked]> wrote:
> Agreed.
> That's a strong requirement that may limit the scalability of the server.
> 
> The spirit of the pNFS operations, at least from Panasas perspective was that
> the data is transient until LAYOUTCOMMIT, meaning it may or may not be visible
> to clients other than the one who wrote it, and its associated metadata MUST
> be updated and describe the new data only on LAYOUTCOMMIT and until then it's
> undefined, i.e. it's up to the server implementation whether to update it or not.
> 
> Without locking, what do the stronger semantics buy you?
> Even if a client verified the change_attribute new data may become visible
> at any time after the GETATTR if the file/byte range aren't locked.

There is no locking needed in the scenario below: it is ordinary
close-to-open semantics.

The point is that if you remove the one and only way that clients have
to determine whether or not their data caches are valid, then they can
no longer cache data at all, and server scalability will be shot to
smithereens anyway.

Trond

> > _______________________________________________
> > nfsv4 mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Dean Hildebrand <[email-masked]>
Date: Wed, 07 Jul 2010 10:45:54 -0700

On 7/7/2010 7:03 AM, Trond Myklebust wrote:
>> On Jul. 07, 2010, 16:18 +0300, Trond Myklebust<[email-masked]>  wrote:
>>      
>>>>> On Jul. 06, 2010, 23:40 +0300, Trond Myklebust<[email-masked]>  wrote:
>>>>>            

It would seem that when the change_attr is changed depends on the server 
implementation.  If the
server implementation promises NOT to modify the file in place on a 
write, then it can postpone
updating the change_attr until LAYOUTCOMMIT (at which time the actual 
file data is updated).  If
not, meaning that if client 1 can see the write by client 2 in the 
example below, then the change_attr
should be updated on every write (I would guess it would only be updated 
when some server actually
requested it)
Dean

---

From: <[email-masked]>
Date: Wed, 7 Jul 2010 16:39:42 -0400

To bring this discussion full circle, since we agree that a compliant
server can implement a scheme where written data does not become visible
until after a LAYOUTCOMMIT, do we also agree that LAYOUTCOMMIT is a
"MUST" from a compliant client (independent of layout type)?

  -Dan

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Trond Myklebust
> Sent: Wednesday, July 07, 2010 7:04 AM
> To: Benny Halevy
> Cc: [email-masked]; [email-masked]; Garth 
> Gibson; Brent Welch; NFSv4
> Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close
>

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 17:01:24 -0400

> "MUST" from a compliant client (independent of layout type)?

Yes. I would agree that the client cannot rely on the updates being made
visible if it fails to send the LAYOUTCOMMIT. My point was simply that a
compliant server MUST also have a valid strategy for dealing with the
case where the client doesn't send it.

Cheers
  Trond

---

From: <[email-masked]>
Date: Wed, 7 Jul 2010 18:04:18 -0400

> case where the client doesn't send it.

So you are saying the updates "MUST be made visible" through the
server's valid strategy.  Is that right.

And that the client cannot rely on that.  Why not, if the server must
have a valid strategy.

Is this just prudent "belt and suspenders" design or what?

It seems to me that if one side here is MUST (and the spec needs to be
clearer about what might or might not constitute a valid strategy), then
the other side should be SHOULD.

If both sides are "MUST", then if things don't work out then the client
and server can equally point to one another and say "It's his fault".

Am I missing something here?

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Trond Myklebust
Sent: Wednesday, July 07, 2010 5:01 PM
To: Muntz, Daniel
Cc: [email-masked]; [email-masked]; [email-masked];
[email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] 4.1 client - LAYOUTCOMMIT & close

Cheers
  Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 18:27:58 -0400

> Am I missing something here?

See the example at the very bottom of this email. If the client dies
after it has written data to the data servers, but before it can issue
LAYOUTCOMMIT, then the server needs to have a strategy for dealing with
that. Either it has to figure out that changes have been made, and to
update the change attribute so that close-to-open cache consistency
works, or it needs to ensure that those changes are not made visible.

A "solution" where the file data changes, but the client can't detect it
is not acceptable.

Trond

---

From: <[email-masked]>
Date: Wed, 7 Jul 2010 18:44:54 -0400

Let me try this ...

A correct client will always send LAYOUTCOMMIT.
Assume that the client is correct.
Hence if the LAYOUTCOMMIT doesn't arrive, something's failed.

Important implication: No LAYOUTCOMMIT is an error/failure case.  It
just has to work; it doesn't have to be fast.

Suggestion: If a client dies while holding writeable layouts that permit
write-in-place, and the client doesn't reappear or doesn't reclaim those
layouts, then the server should assume that the files involved were
written before the client died, and set the file attributes accordingly
as part of internally reclaiming the layout that the client has
abandoned.

Caveat: It may take a while for the server to determine that the client
has abandoned a layout.

This can result in false positives (file appears to be modified when it
wasn't) but won't yield false negatives (file does not appear to be
modified even though it was modified).

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf
Of [email-masked]
> Sent: Wednesday, July 07, 2010 6:04 PM
> To: [email-masked]; Muntz, Daniel
[email-masked];
> [email-masked]; [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 18:52:55 -0400

> modified even though it was modified).

OK... So we're going to have to turn off client side file caching
entirely for pNFS? I can do that...

The above won't work. Think readahead...

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 19:09:09 -0400

> The above won't work. Think readahead...

So... What can work, is if you modify it to work explicitly for
close-to-open

"Upon receiving an OPEN, LOCK or a WANT_DELEGATION, the server must
check that it has received LAYOUTCOMMITs from any other clients that may
have the file open for writing. If it hasn't, then it MUST take some
action to ensure that any file data changes are accompanied by a change
attribute update."

Then you can add the above suggestion without the offending caveat. Note
however that it does break the "SHOULD NOT" admonition in section
18.32.4.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 07 Jul 2010 19:14:57 -0400

> action to ensure that any file data changes are accompanied by a change
                           ^ potentially visible

---

From: Benny Halevy <[email-masked]>
Date: Thu, 08 Jul 2010 18:59:31 +0300

>>>> just has to work; it doesn't have to be fast.

Note that a LAYOUTRETURN can arrive without LAYOUTCOMMIT if the client hasn't
written to the file.  I'm not sure what about the blocks case though, do you
implicitly free up any provisionally allocated blocks that the client had not
explicitly committed using LAYOUTCOMMIT?

>>>> as part of internally reclaiming the layout that the client has

Of course. That's part of the server recovery.

>>>> Caveat: It may take a while for the server to determine that the client

That's two lease times after a respective CB_LAYOUTRECALL.

>> action to ensure that any file data changes are accompanied by a change

That should be OK as long as it's not for every GETATTR for the change, mtime,
or size attributes.

>> however that it does break the "SHOULD NOT" admonition in section

Better be safe than sorry in this rare error case.

Benny

---

From: <[email-masked]>
Date: Thu, 8 Jul 2010 16:30:48 -0400

> explicitly committed using LAYOUTCOMMIT?

In principle, yes as the blocks are no longer promised to the client, although
lazy evaluation of this is an obvious optimization.

> Better be safe than sorry in this rare error case.

I concur with Benny on both of the above - in essence, the unrecovered client failure is a reason to potentially ignore the "SHOULD" (server can't know whether it actually ignored the "SHOULD", hence better safe than sorry).  We probably ought to find a someplace appropriate to add a paragraph or two explaining this in one of the 4.2 documents.

Thanks,
--David

> From: Benny Halevy [mailto:[email-masked]] On Behalf Of Benny Halevy
> Sent: Thursday, July 08, 2010 12:00 PM
> To: Trond Myklebust
> Cc: Black, David; Noveck, David; Muntz, Daniel; [email-masked]; [email-masked];
> [email-masked]; [email-masked]; [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 08 Jul 2010 17:16:11 -0400

> I concur with Benny on both of the above - in essence, the unrecovered client failure is a reason to potentially ignore the "SHOULD" (server can't know whether it actually ignored the "SHOULD", hen…

Right. I'm only interested in fixing the close-to-open case. The case of
general GETATTR calls might be nice to fix too, but it should not be
essential in order to ensure that well-behaved applications continue to
work as expected.

Note, however, that legacy support for stateless protocols like NFSv2
and NFSv3 may be problematic: there is no equivalent of OPEN, and so the
server may have to do the above check on all NFSPROC2_GETATTR,
NFSPROC3_GETATTR, NFSPROC2_LOOKUP and NFSPROC3_LOOKUP requests.

   Trond

---

From: sfaibish <[email-masked]>
Date: Thu, 08 Jul 2010 18:12:47 -0400

All, After discussing this issue with Dave Noveck and as I mentioned in the
call today I think that this is a serious issue and a disconnect between
different layout types behavior. My proposal is to have this discussion F2F
in Maastricht on the white board. So I will add an agenda item to the WG
on this topic. I could address the behavior of the block layout but
it is not something we want to mimic as we all agreed at cthon to avoid the
LAYOUTCOMMIT as much as possible for file layout. If we solve the
issue using the proposed mechanism (Trond) we will create a conflict
with the use of LAYOUTCOMMIT. Just as a hint the difference from block is
that block uses layout for write and read as different leases and
when a client has layout for read the server will always send him
a LAYOUTRETURN when either upgrading his lease to write of send a layout
for write to another client. We don't want to do same for file, I
don't think so. My 2c.

/Sorin

On Thu, 08 Jul 2010 16:30:48 -0400, <[email-masked]> wrote:

> I concur with Benny on both of the above - in essence, the unrecovered  
> client failure is a reason to potentially ignore the "SHOULD" (server  
> can't know whether it actually ignored the "SHOULD", hence better safe  
> than sorry).  We probably ought to find a someplace appropriate to add a  
> paragraph or two explaining this in one of the 4.2 documents.
>
> Thanks,
> --David
>
>
>> Cc: Black, David; Noveck, David; Muntz, Daniel;  
>> [email-masked]; [email-masked];
>> >>>>>> Yes. I would agree that the client cannot rely on the updates  
>> being
>> >>>> made
>> >>>>> clearer about what might or might not constitute a valid  
>> strategy),
>> >>>> then
>> >>>>> Yes. I would agree that the client cannot rely on the updates  
>> being
>> >>>> made
>> >>>>> visible if it fails to send the LAYOUTCOMMIT. My point was simply  
>> that
>> >>>> a
>> >>>>>>> The point is that if you remove the one and only way that  
>> clients
>> >>>> have
>> >>>>>>> to determine whether or not their data caches are valid, then  
>> they
>> >>>> can

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Network Storage Group

         EMC²
where information lives

Phone: 508-435-1000 x 48545
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Tom Haynes <[email-masked]>
Date: Thu, 08 Jul 2010 18:01:16 -0500

> for write to another client. We don't want to do same for file, I

When I hear the words "white board", I immediately think unorganized and 
likely
to get out of hand. I don't know how much time we are up to now, but we must
be close to running out of it.

I have a counter-proposal, why doesn't someone, say Trond, put together
some slides on this and we discuss them.

Or, if there is a strong consensus that we do need to do this on a white
board, why don't we ask ietf for an additional slot in the morning?

---

From: <[email-masked]>
Date: Thu, 8 Jul 2010 19:51:47 -0400

> [mailto:[email-masked]] On Behalf Of Trond Myklebust
> Sent: Thursday, July 08, 2010 2:16 PM
> To: Black, David
> Cc: [email-masked]; [email-masked]; 
> [email-masked]; [email-masked]; [email-masked]; 
> [email-masked]
> > I concur with Benny on both of the above - in essence, the 
> unrecovered client failure is a reason to potentially ignore 
> the "SHOULD" (server can't know whether it actually ignored 
> the "SHOULD", hence better safe than sorry).  We probably 
> ought to find a someplace appropriate to add a paragraph or 
> two explaining this in one of the 4.2 documents.
> 

I think we have close-to-open covered.  A client will do a LAYOUTCOMMIT
after the last WRITE before a CLOSE (otherwise it has no guarantee that
the data becomes "visible").  So, written data may be "visible" to other
clients without the change attribute being updated, *but* at CLOSE time
we are guaranteed the change attribute is updated.

In the failure case (client dies before sending LAYOUTCOMMIT and/or
CLOSE), the server will eventually have to close the file.  At this
point, the server can, e.g., use its knowledge of the layout(s) that may
have been used by the client to check DSs (via control protocol) to
synthesize the appropriate attributes including change attribute and set
them before completing the server close operation.  This is hand-wavy,
but I think there's a way to solve close-to-open without updating the
change attribute with every DS write.

However, I think we may still have a problem when locking/delegations
are combined with client caching and attempting to decouple the DS write
from the change attribute update.  I'm still looking into this.

> > > >>>>> From: [email-masked] 
> [mailto:[email-masked]] On Behalf
> > > >>>> Of [email-masked]
> > > >>>>>> Yes. I would agree that the client cannot rely on 
> the updates being
> > > >>>> made
> > > >>>> that a
> > > >>>>>> compliant server MUST also have a valid strategy 
> for dealing with
> > > >>>> the
> > > >>>>> clearer about what might or might not constitute a 
> valid strategy),
> > > >>>> then
> > > >>>>> If both sides are "MUST", then if things don't work 
> out then the
> > > >>>> client
> > > >>>>> From: [email-masked] 
> [mailto:[email-masked]] On Behalf
> > > >>>>> Of Trond Myklebust
> > > >>>>>> server can implement a scheme where written data 
> does not become
> > > >>>>> visible
> > > >>>>> Yes. I would agree that the client cannot rely on 
> the updates being
> > > >>>> made
> > > >>>>> visible if it fails to send the LAYOUTCOMMIT. My 
> point was simply that
> > > >>>> a
> > > >>>>>>>>>>>> On Tue, 2010-07-06 at 15:20 -0400, 
> [email-masked]
> > > >>>>> wrote:
> > > >>>>>>>>>>>>> The COMMIT to the DS, ttbomk, commits data 
> on the DS. I
> > > >>>> see it as
> > > >>>>>>>>>>>>> As sjoshi@bluearc mentioned, the 
> LAYOUTCOMMIT provides a
> > > >>>> synchronization
> > > >>>>>>>>>>>>> point, so even if the non-clustered server 
> does not want
> > > >>>> to update
> > > >>>>>>>>>>>>> metadata on every DS I/O, the LAYOUTCOMMIT 
> could also be a
> > > >>>> trigger to
> > > >>>>>>>>>>>>> execute whatever synchronization mechanism 
> the implementer
> > > >>>> wishes to put
> > > >>>>>>>>>>>>> in the control protocol.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> As far as I'm aware, there are no exceptions 
> in RFC5661
> > > >>>> that would allow
> > > >>>>>>>>>>>> pNFS servers to break the rule that any 
> visible change to
> > > >>>> the data must
> > > >>>>>>>>>>> See more in section 12.5.4 and 12.5.4.1. 
> LAYOUTCOMMIT and
> > > >>>> change/time_modify
> > > >>>>>>>>>>> in particular:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>    For some layout protocols, the storage 
> device is able to
> > > >>>> notify the
> > > >>>>>>>>>>>    metadata server of the occurrence of an 
> I/O; as a result,
> > > >>>> the change
> > > >>>>>>>>>>>    and time_modify attributes may be updated 
> at the metadata
> > > >>>> server.
> > > >>>>>>>>>>>    required to update the change attribute.  
> In this case,
> > > >>>> the metadata
> > > >>>>>>>>>>>    server must ensure that no further update 
> to the data has
> > > >>>> occurred
> > > >>>>>>>>>>>    have enough information to make this 
> determination or may
> > > >>>> update the
> > > >>>>>>>>>>>    change attribute upon each file 
> modification.  This also
> > > >>>> applies for
> > > >>>>>>>>>>>    the time_modify attribute.  If the server 
> implementation
> > > >>>> is able to
> > > >>>>>>>>>>>    determine that the file has not been 
> modified since the
> > > >>>> last
> > > >>>>>>>>>>>    LAYOUTCOMMIT.  At LAYOUTCOMMIT completion, 
> the updated
> > > >>>> attributes
> > > >>>>>>>>>>>    should be visible if that file was 
> modified since the
> > > >>>> latest previous
> > > >>>>>>>>>> I know. However the above paragraph does not 
> state that the
> > > >>>> server
> > > >>>>>>>>>> should make those changes visible to clients 
> other than the
> > > >>>> one that is
> > > >>>>>>>>>> writing.
> > > >>>>>>>>>>
> > > >>>>>>>>>> change attributes to be updated (if and only 
> if the file data
> > > >>>> is
> > > >>>>>>>>>> modified). Several other sections rely on this 
> behaviour,
> > > >>>> including
> > > >>>>>>>>>> The only 'special behaviour' that I see 
> allowed for pNFS is
> > > >>>> in section
> > > >>>>>>>>>> semantics to work. Again, if this is to be the 
> case, then the
> > > >>>> server
> > > >>>>>>>>>> _must_ be able to deal with the case where 
> client 1 dies
> > > >>>> before it can
> > > >>>>>>>>>> issue the LAYOUTCOMMIT.
> > > >>>>>>>>
> > > >>>>>>>> Agreed.
> > > >>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>>> As I see it, if your server allows one 
> client to read data
> > > >>>> that may have
> > > >>>>>>>>>>>> been modified by another client that holds a 
> WRITE layout
> > > >>>> for that range
> > > >>>>>>>>>>> the requirement for the server in WRITE's 
> implementation
> > > >>>> section
> > > >>>>>>>>>>> is quite weak: "It is assumed that the act of 
> writing data
> > > >>>> to a file will
> > > >>>>>>>>>>> cause the time_modified and change attributes 
> of the file to
> > > >>>> be updated."
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> The difference here is that for pNFS the 
> written data is not
> > > >>>> guaranteed
> > > >>>>>>>>>>> may be visible to other processes on the same 
> host but not
> > > >>>> to others until
> > > >>>>>>>>>>> fsync() or close() - open-to-close semantics 
> are the only
> > > >>>> thing the client
> > > >>>>>>>>>>> guarantees, right?  Issuing LAYOUTCOMMIT on 
> fsync() and
> > > >>>> close() ensure the
> > > >>>>>>>>>>> data is committed to stable storage and is 
> visible to all
> > > >>>> other clients in
> > > >>>>>>>>>>> the cluster.
> > > >>>>>>>>>>
> > > >>>>>>>>>> See above. I'm not disputing your statement 
> that 'the written
> > > >>>> data is
> > > >>>>>>>>>> assumption that 'the written data may be 
> visible without an
> > > >>>> accompanying
> > > >>>>>>>>> In other words, I'd expect the following 
> scenario to give the
> > > >>>> same
> > > >>>>>>>> That's a strong requirement that may limit the 
> scalability of
> > > >>>> the server.
> > > >>>>>>>>
> > > >>>>>>>> the data is transient until LAYOUTCOMMIT, 
> meaning it may or may
> > > >>>> not be visible
> > > >>>>>>>> to clients other than the one who wrote it, and 
> its associated
> > > >>>> metadata MUST
> > > >>>>>>>> be updated and describe the new data only on 
> LAYOUTCOMMIT and
> > > >>>> until then it's
> > > >>>>>>>> undefined, i.e. it's up to the server 
> implementation whether to
> > > >>>> update it or not.
> > > >>>>>>>>
> > > >>>>>>>> Even if a client verified the change_attribute 
> new data may
> > > >>>> become visible
> > > >>>>>>>> at any time after the GETATTR if the file/byte 
> range aren't
> > > >>>> locked.
> > > >>>>>>>
> > > >>>>>>> The point is that if you remove the one and only 
> way that clients
> > > >>>> have
> > > >>>>>>> to determine whether or not their data caches are 
> valid, then they
> > > >>>> can
> > > >>>>>>> no longer cache data at all, and server 
> scalability will be shot
> > > >>>> to
> > > >>>>>>> smithereens anyway.
> > > >>>>>>>
> > > >>>>>>> Trond
> > > >>>>>>>
> > > >>>>>>>> Benny
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Client 1			Client 2
> > > >>>>>>>>> ========			========
> > > >>>>>>>>>
> > > >>>>>>>>> OPEN foo
> > > >>>>>>>>> READ
> > > >>>>>>>>> CLOSE
> > > >>>>>>>>> 				OPEN
> > > >>>>>>>>> 				LAYOUTGET ...
> > > >>>>>>>>> 				WRITE via DS
> > > >>>>>>>>> 				<dies>...
> > > >>>>>>>>> OPEN foo
> > > >>>>>>>>> verify change_attr
> > > >> More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> > > >
> > > >
> > > >
> > > > --
> > > > More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> > > 
> >

---

From: sfaibish <[email-masked]>
Date: Thu, 08 Jul 2010 19:57:37 -0400

> some slides on this and we discuss them.
Agreed. This is what I thought about "white board" a presentation followed  
by
a discussion on plan of action, perhaps a new 4.2 draft if there is a need.
We can continue it in the email after we decide what to do.

> board, why don't we ask ietf for an additional slot in the morning?
My bad using the wrong term. I don't think we need a special time slot
but we can decide on the spot in Maastricht. We should be able to find a
room available.

-- 
Best Regards

Sorin Faibish

         EMC²
where information lives

---

From: "Sandeep Joshi" <[email-masked]>
Date: Thu, 8 Jul 2010 17:03:58 -0700

It seems like we agree that laycommit will be sent for file layout,
correct?
Or Should I file a defect on this?

For reference my original email below.

// START
In certain cases, I don't see layoutcommit on a file at all even after
doing many writes.

Client side operations:

open
write(s)
close

open
layoutget's
close

But, I do not see laycommit at all. In terms data written by client it
is about 4-5MB.

// END

Regards,

Sandeep

-----Original Message-----
To: [email-masked]

On Thu, 2010-07-08 at 16:30 -0400, [email-masked] wrote:
> > Note that a LAYOUTRETURN can arrive without LAYOUTCOMMIT if the 
> > client hasn't written to the file.  I'm not sure what about the 
> > blocks case though, do you implicitly free up any provisionally 
> > allocated blocks that the client had not explicitly committed using
LAYOUTCOMMIT?
> 
> In principle, yes as the blocks are no longer promised to the client, 
> although lazy evaluation of this is an obvious optimization.
> 
> > >> "Upon receiving an OPEN, LOCK or a WANT_DELEGATION, the server 
> > >> must check that it has received LAYOUTCOMMITs from any other 
> > >> clients that may have the file open for writing. If it hasn't, 
> > >> then it MUST take some action to ensure that any file data 
> > >> changes are accompanied by a change
> > >                            ^ potentially visible
> > >> attribute update."
> > 
> > That should be OK as long as it's not for every GETATTR for the 
> > change, mtime, or size attributes.
> > 
> > >>
> > >> Then you can add the above suggestion without the offending 
> > >> caveat. Note however that it does break the "SHOULD NOT" 
> > >> admonition in section 18.32.4.
> > 

   Trond

> > [email-masked]; [email-masked]; [email-masked]; 
> > [email-masked]; [email-masked]
> > >>>> Important implication: No LAYOUTCOMMIT is an error/failure 
> > >>>> case.  It just has to work; it doesn't have to be fast.
> > >>>>
> > 
> > Note that a LAYOUTRETURN can arrive without LAYOUTCOMMIT if the 
> > client hasn't written to the file.  I'm not sure what about the 
> > blocks case though, do you implicitly free up any provisionally 
> > allocated blocks that the client had not explicitly committed using
LAYOUTCOMMIT?
> > 
> > >>>> Suggestion: If a client dies while holding writeable layouts 
> > >>>> that permit write-in-place, and the client doesn't reappear or 
> > >>>> doesn't reclaim those layouts, then the server should assume 
> > >>>> that the files involved were written before the client died, 
> > >>>> and set the file attributes accordingly as part of internally 
> > >>>> reclaiming the layout that the client has abandoned.
> > 
> > >>>> Caveat: It may take a while for the server to determine that 
> > >>>> the client has abandoned a layout.
> > 
> > >>>> This can result in false positives (file appears to be modified

> > >>>> when it
> > >>>> wasn't) but won't yield false negatives (file does not appear 
> > >>>> to be modified even though it was modified).
> > >>>
> > >>> OK... So we're going to have to turn off client side file 
> > >>> caching entirely for pNFS? I can do that...
> > >>>
> > >> "Upon receiving an OPEN, LOCK or a WANT_DELEGATION, the server 
> > >> must check that it has received LAYOUTCOMMITs from any other 
> > >> clients that may have the file open for writing. If it hasn't, 
> > >> then it MUST take some action to ensure that any file data 
> > >> changes are accompanied by a change
> > >                            ^ potentially visible
> > >> attribute update."
> > 
> > That should be OK as long as it's not for every GETATTR for the 
> > change, mtime, or size attributes.
> > 
> > >>
> > >> Then you can add the above suggestion without the offending 
> > >> caveat. Note however that it does break the "SHOULD NOT" 
> > >> admonition in section 18.32.4.
> > 
> > >>>>> On Behalf

> > >>>>>> compliant server MUST also have a valid strategy for dealing 
> > >>>>>> with
> > >>>> the
> > >>>>> So you are saying the updates "MUST be made visible" through 
> > >>>>> the server's valid strategy.  Is that right.
> > >>>>>
> > >>>>> And that the client cannot rely on that.  Why not, if the 
> > >>>>> server must have a valid strategy.
> > >>>>>
> > >>>>> It seems to me that if one side here is MUST (and the spec 
> > >>>>> needs to be clearer about what might or might not constitute a

> > >>>>> If both sides are "MUST", then if things don't work out then 
> > >>>>> the
> > >>>> client
> > >>>>> Cc: [email-masked]; [email-masked]; 
> > >>>>> [email-masked]; [email-masked]; [email-masked]; 
> > >>>>> [email-masked]
> > >>>>>> server can implement a scheme where written data does not 
> > >>>>>> become
> > >>>>> visible
> > >>>>>> until after a LAYOUTCOMMIT, do we also agree that 
> > >>>>>> LAYOUTCOMMIT is a "MUST" from a compliant client (independent
of layout type)?
> > >>>>> visible if it fails to send the LAYOUTCOMMIT. My point was 
> > >>>>> simply that
> > >>>> a
> > >>>>> compliant server MUST also have a valid strategy for dealing 
> > >>>>> with the case where the client doesn't send it.
> > >>>>>
> > >>>>> Cheers
> > >>>>>   Trond
> > >>>>>
> > >>>>>>   -Dan
> > >>>>>>

> > >>>>>>>>>>>>> The COMMIT to the DS, ttbomk, commits data on the DS. 
> > >>>>>>>>>>>>> I
> > >>>> see it as
> > >>>>>>>>>>>>> As sjoshi@bluearc mentioned, the LAYOUTCOMMIT provides

> > >>>>>>>>>>>>> a
> > >>>> synchronization
> > >>>>>>>>>>>>> point, so even if the non-clustered server does not 
> > >>>>>>>>>>>>> want
> > >>>> to update
> > >>>>>>>>>>>>> metadata on every DS I/O, the LAYOUTCOMMIT could also 
> > >>>>>>>>>>>>> be a
> > >>>> trigger to
> > >>>>>>>>>>>>> execute whatever synchronization mechanism the 
> > >>>>>>>>>>>>> implementer
> > >>>> wishes to put
> > >>>>>>>>>>>>> in the control protocol.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> pNFS servers to break the rule that any visible change 
> > >>>>>>>>>>>> to
> > >>>> the data must
> > >>>>>>>>>>> See more in section 12.5.4 and 12.5.4.1. LAYOUTCOMMIT 
> > >>>>>>>>>>> and
> > >>>> change/time_modify
> > >>>>>>>>>>> in particular:
> > >>>>>>>>>>>
> > >>>>>>>>>>>    For some layout protocols, the storage device is able

> > >>>>>>>>>>> to
> > >>>> notify the
> > >>>>>>>>>>>    metadata server of the occurrence of an I/O; as a 
> > >>>>>>>>>>> result,
> > >>>> the change
> > >>>>>>>>>>>    and time_modify attributes may be updated at the 
> > >>>>>>>>>>> metadata
> > >>>> server.
> > >>>>>>>>>>>    required to update the change attribute.  In this 
> > >>>>>>>>>>> case,
> > >>>> the metadata
> > >>>>>>>>>>>    server must ensure that no further update to the data

> > >>>>>>>>>>> has
> > >>>>>>>>>>>    have enough information to make this determination or

> > >>>>>>>>>>> may
> > >>>> update the
> > >>>>>>>>>>>    change attribute upon each file modification.  This 
> > >>>>>>>>>>> also
> > >>>> applies for
> > >>>>>>>>>>>    determine that the file has not been modified since 
> > >>>>>>>>>>> the
> > >>>> last
> > >>>>>>>>>>>    LAYOUTCOMMIT.  At LAYOUTCOMMIT completion, the 
> > >>>>>>>>>>> updated
> > >>>> attributes
> > >>>>>>>>>> I know. However the above paragraph does not state that 
> > >>>>>>>>>> the
> > >>>> server
> > >>>>>>>>>> should make those changes visible to clients other than 
> > >>>>>>>>>> the
> > >>>> one that is
> > >>>>>>>>>> writing.
> > >>>>>>>>>>
> > >>>>>>>>>> change attributes to be updated (if and only if the file 
> > >>>>>>>>>> data
> > >>>> is
> > >>>>>>>>>> The only 'special behaviour' that I see allowed for pNFS 
> > >>>>>>>>>> is
> > >>>> in section
> > >>>>>>>>>> 13.10, which states that clients can't expect to see 
> > >>>>>>>>>> changes immediately, but that they must be able to expect
> > >>>> close-to-open
> > >>>>>>>>>> semantics to work. Again, if this is to be the case, then

> > >>>>>>>>>>>> As I see it, if your server allows one client to read 
> > >>>>>>>>>>>> data
> > >>>> that may have
> > >>>>>>>>>>>> been modified by another client that holds a WRITE 
> > >>>>>>>>>>>> layout
> > >>>> for that range
> > >>>>>>>>>>> is quite weak: "It is assumed that the act of writing 
> > >>>>>>>>>>> data
> > >>>> to a file will
> > >>>>>>>>>>> cause the time_modified and change attributes of the 
> > >>>>>>>>>>> file to
> > >>>> be updated."
> > >>>>>>>>>>>
> > >>>>>>>>>>> The difference here is that for pNFS the written data is

> > >>>>>>>>>>> not
> > >>>>>>>>>>> may be visible to other processes on the same host but 
> > >>>>>>>>>>> not
> > >>>> to others until
> > >>>>>>>>>>> fsync() or close() - open-to-close semantics are the 
> > >>>>>>>>>>> only
> > >>>> thing the client
> > >>>>>>>>>>> data is committed to stable storage and is visible to 
> > >>>>>>>>>>> all
> > >>>> other clients in
> > >>>>>>>>>>> the cluster.
> > >>>>>>>>>>
> > >>>>>>>>>> See above. I'm not disputing your statement that 'the 
> > >>>>>>>>>> written
> > >>>> data is
> > >>>>>>>>>> assumption that 'the written data may be visible without 
> > >>>>>>>>>> an
> > >>>> accompanying
> > >>>>>>>>> In other words, I'd expect the following scenario to give 
> > >>>>>>>>> the
> > >>>> same
> > >>>>>>>> That's a strong requirement that may limit the scalability 
> > >>>>>>>> of
> > >>>> the server.
> > >>>>>>>>
> > >>>>>>>> the data is transient until LAYOUTCOMMIT, meaning it may or

> > >>>>>>>> may
> > >>>> not be visible
> > >>>>>>>> to clients other than the one who wrote it, and its 
> > >>>>>>>> associated
> > >>>> metadata MUST
> > >>>>>>>> be updated and describe the new data only on LAYOUTCOMMIT 
> > >>>>>>>> and
> > >>>> until then it's
> > >>>>>>>> undefined, i.e. it's up to the server implementation 
> > >>>>>>>> whether to
> > >>>> update it or not.
> > >>>>>>>>
> > >>>>>>> There is no locking needed in the scenario below: it is 
> > >>>>>>> ordinary close-to-open semantics.
> > >>>>>>>
> > >>>>>>> to determine whether or not their data caches are valid, 
> > >>>>>>> then they
> > >>>> can
> > >>>>>>> no longer cache data at all, and server scalability will be 
> > >>>>>>> shot
> > >>>> to
> > >>>>>>> smithereens anyway.
> > >>>>>>>
> > >>>>>>> Trond
> > >>>>>>>
> > >>>>>>>> Benny
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Client 1			Client 2
> > >>>>>>>>> ========			========
> > >>>>>>>>>
> > >>>>>>>>> OPEN foo
> > >>>>>>>>> READ
> > >>>>>>>>> CLOSE
> > >>>>>>>>> 				OPEN
> > >>>>>>>>> 				LAYOUTGET ...
> > >>>>>>>>> 				WRITE via DS
> > >>>>>>>>> 				<dies>...
> > >>>>>>>>> OPEN foo
> > >>>>>>>>> verify change_attr
> > >>>>>>>>> READ if above WRITE is visible CLOSE
> > >>>>>>>>>
> > >>>>>>>>> Trond
> > >> linux-nfs" in the body of a message to [email-masked] 
> > >> More majordomo info at  
> > > linux-nfs" in the body of a message to [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 08 Jul 2010 20:41:01 -0400

> some slides on this and we discuss them.

Say who, what???? :-)

OK. I can put something together, but it will take 5-10 minutes of
meeting time (10 being the more realistic estimate).

  Trond