Subject: [nfsv4] Write-behind caching

From: <[email-masked]>
Date: Mon, 25 Oct 2010 21:58:07 -0400

The following statement appears at the bottom of page 292 of RFC5661.  

   However, write-behind caching may negatively
   affect the latency in returning a layout in response to a
   CB_LAYOUTRECALL; this is similar to file delegations and the impact
   that file data caching has on DELEGRETURN.  Client implementations
   SHOULD limit the amount of unwritten data they have outstanding at
   any one time in order to prevent excessively long responses to
   CB_LAYOUTRECALL.

This does not seem to make sense to me.

First of all the analogy between DELEGRETURN and
CB_LAYOUTRECALL/LAYOUTRETURN doesn't seem to me to be correct.  In the
case of DELEGRETURN, at least if the file in question has been closed,
during the pendency of the delegation, you do need to write all of the
dirty data associated with those previously open files.  Normally,
clients just write all dirty data.

LAYOUTRETURN does not have that sort of requirement.  If it is valid to
hold the dirty data when you do have the layout, it is just as valid to
hold it when you don't.  You could very well return the layout and get
it again before some of those dirty blocks are written.  Having a layout
grants you the right to do IO using a particular means (different based
on the mapping type), but if you don't have the layout, you still have a
way to do the writeback, and there is no particular need to write back
all the data before returning the layout.  As mentioned above, you may
well get the layout again before there is any need to actually do the
write-back.

You have to wait until IO's that are in flight are completed before you
return the layout.  However, I don't see why you would have to or want
to start new IO's using the layout if you have received a
CB_LAYOUTRECALL..

Am I missing something?  Is there some valid reason for this statement?
Or should this be dealt with via the errata mechanism?

What do existing clients actually do with pending writeback data when
they get a CB_LAYOUTRECALL?  Do they start new IO's using the layout?
If so, is there any other reason other than the paragraph above?

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 26 Oct 2010 02:06:52 +0000

Since this description is part of the general pNFS description, the
intent may have been to cover a variety of layout types.  However,
I agree that the client is not guaranteed access to the layout and
is fully capable of writing the data via the MDS if all else
fails (inability to obtain the layout after a return); it may not
be the most performant path but it should be functional.  And maybe
that is the source of the statement that the client should take
care in managing its dirty pages given the lack of guarantee of
access to the supposed, higher throughput path for writing data.

As implementation guidance it seems okay but truly a requirement
for correct function.

Spencer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> [email-masked]
> Sent: Monday, October 25, 2010 6:58 PM
> To: [email-masked]
> Subject: [nfsv4] Write-behind caching
> 
> dirty data associated with those previously open files.  Normally, clients
> just write all dirty data.
> 
> hold it when you don't.  You could very well return the layout and get it
> again before some of those dirty blocks are written.  Having a layout
> grants you the right to do IO using a particular means (different based on
> the mapping type), but if you don't have the layout, you still have a way
> to do the writeback, and there is no particular need to write back all the
> data before returning the layout.  As mentioned above, you may well get
> the layout again before there is any need to actually do the write-back.
> 
> return the layout.  However, I don't see why you would have to or want to
> start new IO's using the layout if you have received a CB_LAYOUTRECALL..
> 
> What do existing clients actually do with pending writeback data when they
> get a CB_LAYOUTRECALL?  Do they start new IO's using the layout?
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Mon, 25 Oct 2010 23:24:35 -0400

I agree that the intent was to cover a variety of layout types.

I think what you are saying about the issue of different throughputs for
having and not having layouts also makes sense.  It may in some way have
led to the statement in RFC5661 but those statements are by no means the
same.  They have different consequences.  I take it that you are saying
(correctly) something like:

     However, write-behind implementations will generally need to bound
     the amount of unwritten date so that given the bandwidth of the 
     output path, the data can be written in a reasonable time.  Clients

     which have layouts should avoid keeping larger amounts to reflect a
     situation in which a layout provides a write path of higher
bandwidth.
     This is because a CB_LAYOUTRECALL may be received.  The client
     should not delay returning the layout so as to use that
higher-bandwidth
     path, so it is best if it assumes, in limiting the amount of data
     to be written, that the write bandwidth is only what is available
     without the layout, and that it uses this bandwidth assumption even
     if it does happen to have a layout.

This differs from the text in RFC5661 in a few respects.

	First it says that the amount of dirty data should be the same
when
	you have the layout and when you don't, rather than simply
saying it
	should be small when you have the layout, possibly implying that
it
 	should be smaller than when you don't have a layout.

	Second the text now in RFC5661 strongly implies that when you
get
	CB_LAYOUTRECALL, you would normally start new IO's, rather than 
      simply drain the pending IO's and return the layout ASAP. 

So I don't agree that what is in RFC5661 is good implementation advice,
particularly in suggesting that clients should delay the LAYOUTRETURN
while doing a bunch of IO, including starting new IO's.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Spencer Shepler
Sent: Monday, October 25, 2010 10:07 PM
To: Noveck, David; [email-masked]
Subject: Re: [nfsv4] Write-behind caching

Spencer

> dirty data associated with those previously open files.  Normally,
clients
> hold it when you don't.  You could very well return the layout and get
it
> return the layout.  However, I don't see why you would have to or want
to
> What do existing clients actually do with pending writeback data when
they

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 26 Oct 2010 03:34:18 +0000

Fair enough.  I haven't looked to see if the layout types
address this specific, needed, behavior.  Obviously the
statement you reference and the individual layout descriptions
should be tied together.  Again, I don't remember but there
may be layout specific steps needed in the case of handling
layoutreturns.  

In any case, we can handle the eventual conclusion as an errata.

Spencer

> From: [email-masked] [mailto:[email-masked]]
> Sent: Monday, October 25, 2010 8:25 PM
> To: Spencer Shepler; [email-masked]
> Subject: RE: [nfsv4] Write-behind caching
> 
>      should not delay returning the layout so as to use that higher-
> bandwidth
> 	First it says that the amount of dirty data should be the same when
> 	you have the layout and when you don't, rather than simply saying it
> 	should be small when you have the layout, possibly implying that it
> 	Second the text now in RFC5661 strongly implies that when you get
> Spencer Shepler
> Since this description is part of the general pNFS description, the intent
> may have been to cover a variety of layout types.  However, I agree that
> the client is not guaranteed access to the layout and is fully capable of
> writing the data via the MDS if all else fails (inability to obtain the
> layout after a return); it may not be the most performant path but it
> should be functional.  And maybe that is the source of the statement that
> the client should take care in managing its dirty pages given the lack of
> guarantee of access to the supposed, higher throughput path for writing
> data.
> 
> As implementation guidance it seems okay but truly a requirement for
> correct function.
> 
> Spencer
> 
> Of

---

From: <[email-masked]>
Date: Tue, 26 Oct 2010 06:34:05 -0400

That makes sense.  Let me take on this issue with regard to the file
layout.  Are there volunteers to address it with regard to block and
object?  It would be great if we could get together in Beijing, discuss
this, and come to a joint conclusion to present to the working group
(via email I mean).  I'm not planning trying to do this before the
working group meeting.  In any case, I'm pretty sure there won't be any
time during the working group meeting. 

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Monday, October 25, 2010 11:34 PM
To: Noveck, David; [email-masked]
Subject: RE: [nfsv4] Write-behind caching

Spencer

when
saying it
it
get
Of
> Since this description is part of the general pNFS description, the
intent
> > the mapping type), but if you don't have the layout, you still have
a
> way
> > to do the writeback, and there is no particular need to write back
all
> the

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 26 Oct 2010 22:07:58 -0700

I remember at one time there was a thought that all dirty data would 
have to be written to disk when it receives a layoutrecall.  Once the 
data was written, it would send a layoutreturn.  I think this was the 
thinking before all the timing issues and other such things cropped up.  
I assume someone wrote that as general advice, somehow thinking that 
responding to a layoutrecall was more important than actually achieving 
good write performance.

In this light, the analogy with delegreturn makes sense if you take a 
very specific example, but obviously not in general.

I would vote to just cut this text, as I think it is simply outdated.
Dean

---

From: Jason Glasgow <[email-masked]>
Date: Wed, 27 Oct 2010 09:05:22 -0400

With regards to block layout there are some specific implementation issues
that historically made it important to flush dirty data before returning a
layout.  Let me describe one that arose when implementing MPFS at EMC.  I
have not followed the UNIX implementations of the pNFS block layout recently
enough to know if this is still a concern.

Assume a page cache with page size 4K, a file system with block size 8K and
a file that is 8K long.  In many UNIX like operating systems NFS3/MPFS
implementations, a 2K write to the range 2K-4K would obtain a layout for
reading and writing, read the first 8K of the file into 2 pages in the
kernel page cache.  It would then copy the 2K of data from userspace and
overwrite the later half of the first page.  When it was time to write out
the page, MPFS would write out the complete 8K block.  If a layout were
recalled between the time the data was read from disk, and when it was
written, it is possible that another client would modify the range of the
file from 0K to 2K.  Unless specific care was taken when flushing the entire
8K block later, data corruption would occur.

There are two ways to avoid the problem.

1. Keep track of the bytes ranges of a page (or file system block) that are
dirty and only perform the read-modify-write cycle while the client holds
the layout.  This can get messy if a clients writes every other byte on a
page.
2.  Do not return a layout containing a dirty page until that page has been
written.

Perhaps this sheds some light on the original motivation.

Regarding the block layout, I am entirely sympathetic to arguments that a
layout recall should only wait for outstanding writes complete, and should
not cause the client to initiate new writes.

-Jason

On Wed, Oct 27, 2010 at 1:07 AM, Dean Hildebrand <[email-masked]>wrote:

> I remember at one time there was a thought that all dirty data would have
> to be written to disk when it receives a layoutrecall.  Once the data was
> written, it would send a layoutreturn.  I think this was the thinking before
> all the timing issues and other such things cropped up.  I assume someone
> wrote that as general advice, somehow thinking that responding to a
> layoutrecall was more important than actually achieving good write
> performance.
>
> In this light, the analogy with delegreturn makes sense if you take a very
> specific example, but obviously not in general.
>
>>> I think what you are saying about the issue of different throughputs
>>>
>> for
>>
>>> having and not having layouts also makes sense.  It may in some way
>>>
>> have
>>
>>> led to the statement in RFC5661 but those statements are by no means
>>>
>> the
>>
>>> same.  They have different consequences.  I take it that you are
>>>
>> saying
>>
>>>      However, write-behind implementations will generally need to
>>>
>> bound
>>
>>>      output path, the data can be written in a reasonable time.
>>>
>> Clients
>>
>>>      which have layouts should avoid keeping larger amounts to reflect
>>>
>> a
>>
>>>      without the layout, and that it uses this bandwidth assumption
>>>
>> even
>>
>>> So I don't agree that what is in RFC5661 is good implementation
>>>
>> advice,
>>
>>> may have been to cover a variety of layout types.  However, I agree
>>>
>> that
>>
>>> the client is not guaranteed access to the layout and is fully capable
>>>
>> of
>>
>>> writing the data via the MDS if all else fails (inability to obtain
>>>
>> the
>>
>>> should be functional.  And maybe that is the source of the statement
>>>
>> that
>>
>>> the client should take care in managing its dirty pages given the lack
>>>
>> of
>>
>>> guarantee of access to the supposed, higher throughput path for
>>>
>> writing
>>
>>> data.
>>>
>>>> From: [email-masked] [mailto:[email-masked]] On
>>>>
>>> Behalf
>>
>>> Of
>>>
>>>> [email-masked]
>>>> The following statement appears at the bottom of page 292 of
>>>>
>>> RFC5661.
>>
>>>>    CB_LAYOUTRECALL; this is similar to file delegations and the
>>>>
>>> impact
>>
>>>    that file data caching has on DELEGRETURN.  Client
>>>>
>>> implementations
>>
>>>    SHOULD limit the amount of unwritten data they have outstanding
>>>>
>>> at
>>
>>>> CB_LAYOUTRECALL/LAYOUTRETURN doesn't seem to me to be correct.  In
>>>>
>>> the
>>
>>> case of DELEGRETURN, at least if the file in question has been
>>>>
>>> closed,
>>
>>> during the pendency of the delegation, you do need to write all of
>>>>
>>> the
>>
>>>> LAYOUTRETURN does not have that sort of requirement.  If it is valid
>>>>
>>> to
>>>
>>>> hold the dirty data when you do have the layout, it is just as valid
>>>>
>>> to
>>>
>>>> hold it when you don't.  You could very well return the layout and
>>>>
>>> get
>>
>>> it
>>>
>>>> again before some of those dirty blocks are written.  Having a
>>>>
>>> layout
>>
>>> grants you the right to do IO using a particular means (different
>>>>
>>> based on
>>>
>>>> data before returning the layout.  As mentioned above, you may well
>>>>
>>> get
>>>
>>>> the layout again before there is any need to actually do the
>>>>
>>> write-back.
>>>
>>>> You have to wait until IO's that are in flight are completed before
>>>>
>>> you
>>>
>>>> return the layout.  However, I don't see why you would have to or
>>>>
>>> want
>>
>>> to
>>>
>>>> start new IO's using the layout if you have received a
>>>>
>>> CB_LAYOUTRECALL..
>>>
>>>> Am I missing something?  Is there some valid reason for this
>>>>
>>> statement?
>>>
>>>> What do existing clients actually do with pending writeback data
>>>>
>>> when
>>
>>> they
>>>

---

From: Trond Myklebust <[email-masked]>
Date: Wed, 27 Oct 2010 09:40:21 -0400

On Wed, 2010-10-27 at 09:05 -0400, Jason Glasgow wrote:
> With regards to block layout there are some specific implementation
> issues that historically made it important to flush dirty data before
> returning a layout.  Let me describe one that arose when implementing
> MPFS at EMC.  I have not followed the UNIX implementations of the pNFS
> block layout recently enough to know if this is still a concern.
> 
> 
> Assume a page cache with page size 4K, a file system with block size
> 8K and a file that is 8K long.  In many UNIX like operating systems
> NFS3/MPFS implementations, a 2K write to the range 2K-4K would obtain
> a layout for reading and writing, read the first 8K of the file into 2
> pages in the kernel page cache.  It would then copy the 2K of data
> from userspace and overwrite the later half of the first page.  When
> it was time to write out the page, MPFS would write out the complete
> 8K block.  If a layout were recalled between the time the data was
> read from disk, and when it was written, it is possible that another
> client would modify the range of the file from 0K to 2K.  Unless
> specific care was taken when flushing the entire 8K block later, data
> corruption would occur.
>
> 1. Keep track of the bytes ranges of a page (or file system block)
> that are dirty and only perform the read-modify-write cycle while the
> client holds the layout.  This can get messy if a clients writes every
> other byte on a page.
> 2.  Do not return a layout containing a dirty page until that page has
> been written.

3. In order to ensure you can safely do read-modify-write in the block
pNFS case, never request layout sizes that are not a multiple of the
block size.

> Perhaps this sheds some light on the original motivation.

Yes, but the problem you are describing does not justify a _requirement_
that the client flush out its data. Only that it consider the
implications of corner cases such as the above.

Trond

> Regarding the block layout, I am entirely sympathetic to arguments
> that a layout recall should only wait for outstanding writes complete,
> and should not cause the client to initiate new writes.
> 
> 
> -Jason
> 
> 
> 
> 
> On Wed, Oct 27, 2010 at 1:07 AM, Dean Hildebrand
> <[email-masked]> wrote:
>         I remember at one time there was a thought that all dirty data
>         would have to be written to disk when it receives a
>         layoutrecall.  Once the data was written, it would send a
>         layoutreturn.  I think this was the thinking before all the
>         timing issues and other such things cropped up.  I assume
>         someone wrote that as general advice, somehow thinking that
>         responding to a layoutrecall was more important than actually
>         achieving good write performance.
>         
>         In this light, the analogy with delegreturn makes sense if you
>         take a very specific example, but obviously not in general.
>         
>                         From: [email-masked]
>                         [mailto:[email-masked]] On Behalf
>                 Of
>                         Spencer Shepler

---

From: Jason Glasgow <[email-masked]>
Date: Wed, 27 Oct 2010 10:28:09 -0400

I agree with Trond.  This does not justify a requirement. -Jason

On Wed, Oct 27, 2010 at 9:40 AM, Trond Myklebust <[email-masked]
> wrote:

---

From: <[email-masked]>
Date: Wed, 27 Oct 2010 10:52:00 -0400

Trond's item 3 seems good to me but I'm not understanding some
assumptions that are being made.

So suppose you have these partially updated blocks where you are
writing.  It to me to be that it is OK to write in various ways under
the following conditions.

1) You have the block layout not recalled:

     a) It is OK to do an NFS WRITE of a partial block.

     b) It is dangerous to do a block write because of the 
     possibility of corruption.

     c) It would be foolish/erroneous to do an NFS WRITE of
     the full block including the modified part and the part
     read earlier.

2) When the block layout is being recalled:

     a) It is still OK to do an NFS WRITE of the partial 
     block.

     b) It is dangerous to do a block write because of the 
     possibility of corruption. 

     c) It would be foolish/erroneous to do an NFS WRITE of
     the full block including the modified part and the part
     read earlier.

3) When the block layout is not present

     a) It is definitely OK to do an NFS WRITE of the partial 
     block.

     b) Block write is not OK. 

     c) It would be foolish/erroneous to do an NFS WRITE of
     the full block including the modified part and the part
     read earlier.

So to me the message is that when you have partial writes in your buffer
cache you should keep track of them (small bit mask in these cases) and
do the partial write.  When you give up your layout, you retain the
right do the partial NFS WRITE.

So it sounds to me there are two cases.

If having the delegation gives you exclusive access (as if you had a
delegation), then (2b) causes a possible delay problem and the
appropriate warning is to avoid gathering up too many of those and
delaying recalls only applies to partial writes.

On the other hand, if it doesn't, then (1b) and (2b) are corruption
sources anyway and that rather than do (2b), the important thing is the
write when actually done be a (3a) rather than a (3b).

I think the fundamental point is the one others have been making, which
is that there is a difference between, 

     "If there are dirty blocks where specific 
     circumstances, make it advisable that they be 
     written before returning a layout, care should 
     be taken to avoid a significant accumulation of such
     blocks which might unduly delay a pending recall 
     of a layout.  If possible, the client should ensure 
     that the writes can safely be done after the recall 
     is completed, so as avoid this sort of delay."

And

     "If there are dirty blocks, they have to be written
     before returning a layout.  Care should be taken to 
     avoid an undue accumulation of dirty blocks since
     writing these before return the layout might unduly 
     delay a pending recall of layout."

With the first being more complicated and detailed but also accurate.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Trond Myklebust
Sent: Wednesday, October 27, 2010 9:40 AM
To: Jason Glasgow
Cc: [email-masked]
Subject: Re: [nfsv4] Write-behind caching

Trond

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Oct 2010 18:27:38 +0200

> while doing a bunch of IO, including starting new IO's.

That what clora_changed is for.
It's up to the server to provide this hint to the client
and it's up to the client to throttle its dirty cache write-behind in response
to CB_LAYOUTRECALL.
If in your implementation flushing dirty data always seem to be a bad idea
the server can just always set clora_changed to true (though the hint name
is somewhat too specific for the implied semantics)

Benny

---

From: sfaibish <[email-masked]>
Date: Wed, 27 Oct 2010 12:35:44 -0400

>      delay a pending recall of layout."
I can see an additional problem with this statement; depending on the  
amount
of dirty pages that are written before returning the layout the server  
might
get nervous (timeout pending) and either resend the layoutrecall or simply
start fencing the I/Os to DS regardless of the amount of dirty pages
accumulated at the client and that the client expects to write before
returning the layout. So, we might want to address to this problem
in the recommendation.

/Sorin

-- 
Best Regards

Sorin Faibish
Corporate Distinguished Engineer
Unified Storage Division
         EMC²
where information lives

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: <[email-masked]>
Date: Wed, 27 Oct 2010 12:41:29 -0400

I thought I did earlier by saying that you should only keep enough dirty data to be appropriate to the bandwidth you would have without the layout, and, therefore you should only wrap up IO's in progress.  In particular there is no reason to write dirty data where the writes were not actually in flight at the time of the recall.  Wait for the IO to finish; don't start new IO on the layout regardless of the dirty state of pages; return the layout with relative alacrity.  

-----Original Message-----
From: sfaibish [mailto:[email-masked]] 
Sent: Wednesday, October 27, 2010 12:36 PM
To: Noveck, David; [email-masked]; [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Write-behind caching

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Oct 2010 18:48:21 +0200

> returning the layout. So, we might want to address to this problem

Note that this risk exists regardless of flushing any data.
Even when waiting on outstanding I/Os, the amount of I/O
in flight can be large, interconnect can be slow, timeouts can occur, etc.

Benny

---

From: sfaibish <[email-masked]>
Date: Wed, 27 Oct 2010 13:18:37 -0400

> in flight can be large, interconnect can be slow, timeouts can occur,  
My point was to recommend ballanced approach and judjement on client side
to minimize this opportunity of scalability impact. I would leave to
the discretion of the client implementors to find the right ballance
for the writes not in fly. For the infly it is what it is we cannot do
much but for the case Dave was talking about for not in fly. And I
saw what Dave argument was correct still a recomendation for client
implementors might worth beig added.

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: <[email-masked]>
Date: Fri, 29 Oct 2010 13:20:28 -0400

There are two issues here with regard to handling of layout recall.

One is with regard to in-flight IO.  As Benny points out, you cannot be sure that the in-flight IO can be completed in time to avoid the MDS losing patience.  That should rarely be the case though, if things are working right.  The client has to be prepared to deal with IO failures due to layout revocation.  Any IO that was in flight and failed because of layout revocation will need to be handled by being reissued to the MDS.  Is there anybody that disagrees with that?

The second issue concerns IO not in-flight (in other words, not IO's yet but potential IO's) when the recall is received.  I just don't see that it reasonable to start IO's using layout segments being recalled (whether for dirty buffers or anything else).  Doing IO's to the MDS is fine but there is no real need for the layout recall to specially trigger them, whether clora_changed is set or not.

There apparently is a disagreement about this latter point and there are a number of pieces of the spec which assume that, you will or should start IO of dirty blocks in your buffer cache and issue it using a layout in the process of being recalled.  I can't understand the logic behind this.   It is true that that waiting for in-flight IO only is not an absolute guarantee of success, but to me that doesn't argue that somehow it is OK to make the situation even worse by starting IO's using a layout that is being recalled.  There is no way the DS can tell that you didn't issue these a microsecond before you got the recall, so it should work (except for the timing issue as far as return) but I just don't see the spec suggesting that this (starting layout IO using a recalled layout) is a reasonable way for the client to work. 

It appears that there are some people who disagree about this so tell me why I'm wrong.

-----Original Message-----
From: sfaibish [mailto:[email-masked]] 
Sent: Wednesday, October 27, 2010 1:19 PM
To: Benny Halevy
Cc: Noveck, David; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Write-behind caching

/Sorin

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 29 Oct 2010 13:39:55 -0400

> The second issue concerns IO not in-flight (in other words, not IO's yet but potential IO's) when the recall is received.  I just don't see that it reasonable to start IO's using layout segments be…

This should be _very_ rare. Any cases where 2 clients are trying to do
conflicting I/O on the same data is likely to be either a violation of
the NFS cache consistency rules, or a scenario where it is in any case
more efficient to go through the MDS (e.g. writing to adjacent records
that share the same extent).

> There apparently is a disagreement about this latter point and there are a number of pieces of the spec which assume that, you will or should start IO of dirty blocks in your buffer cache and issue…

It isn't...

---

From: sfaibish <[email-masked]>
Date: Fri, 29 Oct 2010 17:32:15 -0400

On Fri, 29 Oct 2010 13:39:55 -0400, Trond Myklebust  
<[email-masked]> wrote:

>> One is with regard to in-flight IO.  As Benny points out, you cannot be  
>> sure that the in-flight IO can be completed in time to avoid the MDS  
>> losing patience.  That should rarely be the case though, if things are  
>> working right.  The client has to be prepared to deal with IO failures  
>> due to layout revocation.  Any IO that was in flight and failed because  
>> of layout revocation will need to be handled by being reissued to the  
>> MDS.  Is there anybody that disagrees with that?
>>
>> The second issue concerns IO not in-flight (in other words, not IO's  
>> yet but potential IO's) when the recall is received.  I just don't see  
>> that it reasonable to start IO's using layout segments being recalled  
>> (whether for dirty buffers or anything else).  Doing IO's to the MDS is  
>> fine but there is no real need for the layout recall to specially  
>> trigger them, whether clora_changed is set or not.
>
Well this is a different discussion: what was the reason for the recall in
the first place. This is one usecase but there could be other usecases
for the recall and we discuss here how to implement the protcol more than
how to solve a real problem. My 2c

/Sorin

>
>> There apparently is a disagreement about this latter point and there  
>> are a number of pieces of the spec which assume that, you will or  
>> should start IO of dirty blocks in your buffer cache and issue it using  
>> a layout in the process of being recalled.  I can't understand the  
>> logic behind this.   It is true that that waiting for in-flight IO only  
>> is not an absolute guarantee of success, but to me that doesn't argue  
>> that somehow it is OK to make the situation even worse by starting IO's  
>> using a layout that is being recalled.  There is no way the DS can tell  
>> that you didn't issue these a microsecond before you got the recall, so  
>> it should work (except for the timing issue as far as return) but I  
>> just don't see the spec suggesting that this (starting layout IO using  
>> a recalled layout) is a reasonable way for the client to work.
>
> It isn't...
>
>> >>>>                 object?  It would be great if we could get  
>> together in
>> >>>>                 Beijing, discuss
>> >>>>                 this, and come to a joint conclusion to present to  
>> the
>> >>>>                 working group
>> >>>>                 working group meeting.  In any case, I'm pretty  
>> sure
>> >>>>                 there won't be any
>> >>>>                 should be tied together.  Again, I don't remember  
>> but
>> >>>>                 there
>> >>>>                 In any case, we can handle the eventual conclusion  
>> as
>> >>>>                 an errata.
>> >>>>
>> >>>>                 Spencer
>> >>>>
>> >>>>
>> >>>>                         I agree that the intent was to cover a  
>> variety
>> >>>>                         of layout types.
>> >>>>
>> >>>>                         I think what you are saying about the  
>> issue of
>> >>>>                         different throughputs
>> >>>>                 for
>> >>>>                              output path, the data can be written  
>> in a
>> >>>>                         reasonable time.
>> >>>>                 Clients
>> >>>>                              which have layouts should avoid  
>> keeping
>> >>>>                         larger amounts to reflect
>> >>>>                 a
>> >>>>                              This is because a CB_LAYOUTRECALL may  
>> be
>> >>>>                         received.  The client
>> >>>>                              should not delay returning the layout  
>> so
>> >>>>                         as to use that higher-
>> >>>>                         bandwidth
>> >>>>                              to be written, that the write  
>> bandwidth
>> >>>>                         is only what is available
>> >>>>                              without the layout, and that it uses  
>> this
>> >>>>                         bandwidth assumption
>> >>>>                 even
>> >>>>                         This differs from the text in RFC5661 in a  
>> few
>> >>>>                         respects.
>> >>>>
>> >>>>                                First it says that the amount of  
>> dirty
>> >>>>                         data should be the same
>> >>>>                 when
>> >>>>                                you have the layout and when you  
>> don't,
>> >>>>                         rather than simply
>> >>>>                 saying it
>> >>>>                                should be smaller than when you  
>> don't
>> >>>>                         have a layout.
>> >>>>
>> >>>>                                Second the text now in RFC5661  
>> strongly
>> >>>>                         implies that when you
>> >>>>                 get
>> >>>>                               simply drain the pending IO's and  
>> return
>> >>>>                         the layout ASAP.
>> >>>>
>> >>>>                         particularly in suggesting that clients  
>> should
>> >>>>                         delay the LAYOUTRETURN
>> >>>>                         while doing a bunch of IO, including  
>> starting
>> >>>>                         new IO's.
>> >>>>
>> >>>>
>> >>>>                         Since this description is part of the  
>> general
>> >>>>                         pNFS description, the
>> >>>>                 intent
>> >>>>                         writing the data via the MDS if all else  
>> fails
>> >>>>                         (inability to obtain
>> >>>>                 the
>> >>>>                         layout after a return); it may not be the  
>> most
>> >>>>                         performant path but it
>> >>>>                         should be functional.  And maybe that is  
>> the
>> >>>>                         source of the statement
>> >>>>                 that
>> >>>>                         As implementation guidance it seems okay  
>> but
>> >>>>                         truly a requirement for
>> >>>>                         correct function.
>> >>>>
>> >>>>                         Spencer
>> >>>>
>> >>>>                                 The following statement appears at  
>> the
>> >>>>                                 bottom of page 292 of
>> >>>>                 RFC5661.
>> >>>>                                    However, write-behind caching  
>> may
>> >>>>                                 negatively
>> >>>>                                    affect the latency in returning  
>> a
>> >>>>                                 layout in response to a
>> >>>>                                    CB_LAYOUTRECALL; this is  
>> similar to
>> >>>>                                 file delegations and the
>> >>>>                 impact
>> >>>>                                    SHOULD limit the amount of
>> >>>>                                 unwritten data they have  
>> outstanding
>> >>>>                 at
>> >>>>                                 CB_LAYOUTRECALL/LAYOUTRETURN  
>> doesn't
>> >>>>                                 seem to me to be correct.  In
>> >>>>                 the
>> >>>>                                 case of DELEGRETURN, at least if  
>> the
>> >>>>                                 file in question has been
>> >>>>                 closed,
>> >>>>                                 during the pendency of the  
>> delegation,
>> >>>>                                 you do need to write all of
>> >>>>                 the
>> >>>>                                 LAYOUTRETURN does not have that  
>> sort
>> >>>>                                 of requirement.  If it is valid
>> >>>>                         to
>> >>>>                                 hold the dirty data when you do  
>> have
>> >>>>                                 the layout, it is just as valid
>> >>>>                         to
>> >>>>                                 grants you the right to do IO  
>> using a
>> >>>>                                 particular means (different
>> >>>>                         based on
>> >>>>                                 to do the writeback, and there is  
>> no
>> >>>>                                 particular need to write back
>> >>>>                 all
>> >>>>                         the
>> >>>>                                 data before returning the layout.   
>> As
>> >>>>                                 mentioned above, you may well
>> >>>>                         get
>> >>>>                                 the layout again before there is  
>> any
>> >>>>                                 need to actually do the
>> >>>>                         write-back.
>> >>>>                                 You have to wait until IO's that  
>> are
>> >>>>                                 in flight are completed before
>> >>>>                         you
>> >>>>                                 return the layout.  However, I  
>> don't
>> >>>>                                 see why you would have to or
>> >>>>                 want
>> >>>>                         to
>> >>>>                                 start new IO's using the layout if  
>> you
>> >>>>                                 have received a
>> >>>>                         CB_LAYOUTRECALL..
>> >>>>                                 Am I missing something?  Is there  
>> some
>> >>>>                                 valid reason for this
>> >>>>                         statement?
>> >>>>                                 Or should this be dealt with via  
>> the
>> >>>>                                 errata mechanism?
>> >>>>
>> >>>>                                 What do existing clients actually  
>> do
>> >>>>                                 with pending writeback data
>> >>>>                 when
>> >>>>                         they
>> >>>>                                 get a CB_LAYOUTRECALL?  Do they  
>> start
>> >>>>                                 new IO's using the layout?
>> >>>>                                 If so, is there any other reason  
>> other
>> >>>>                                 than the paragraph above?
>> >>>>

-- 
Best Regards

Sorin Faibish

Phone: 508-249-5745
Cellphone: 617-510-0422
Email : [email-masked]

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 29 Oct 2010 17:48:53 -0400

> how to solve a real problem. My 2c

I strongly disagree. If this is an unrealistic scenario, then we don't
have to care about devising an optimal strategy for it. The 'there could
be other usecases' scenario needs to be fleshed out before we can deal
with it.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 29 Oct 2010 18:03:43 -0400

> be other usecases' scenario needs to be fleshed out before we can deal

To clarify a bit what I mean: we MUST devise optimal strategies for
realistic and useful scenarios. It is entirely OPTIONAL to devise
optimal strategies for unrealistic ones.

If writing back all data before returning the layout causes protocol
issues because the server cannot distinguish between a bad client and
one that is waiting for I/O to complete, then my argument is that we're
in the second case: we don't have to optimise for it, and so it is safe
for the server to assume 'bad client'...

   Trond