Subject: RE: [nfsv4] pnfs: lingering writes

From: [email-masked]
Date: Thu, 22 Jun 2006 11:48:34 -0400

> In light of David's objection, let me propose the following
> edited version of Garth's paragraph.  I note that Garth's goal
> was to condense my paragraph but the following is certainly
> not condensed.  C'est la vie.

And it's partly my fault for not paying enough attention earlier ...

> Within pNFS, it is possible that write requests may be presented
> to a data server no longer allowed to perform them.  Whether this
> possibility can arise in practice depends on the specific storage
> protocol.  pNFS with a given storage protocol may simply prohibit
> this behavior by mandating that layout not be returned as long
> as there are outstanding writes to the area designated by that layout
> making the client responsible for preventing the situation from
> arising.

I think the latter sentence needs to be split to cover two cases:
- pNFS limits this behavior by requiring a client MUST wait for
	completion of all writes covered by a layout range before
	returning that range.
- pNFS with a specific storage protocol may require (in the
	specification of the layout for use with that protocol)
	that there be no possibility of outstanding duplicate
	write requests when a layout is returned.
The former is something that all pNFS clients have to do.  The
latter sentence allows the block and object layouts to enforce
a stronger requirement (which their protocols make possible).

> For storage protocols where such a mandate is not enforced (including
> use of NFSV4.1 itself as the storage protocol) implementations must
> protect against the possibility that writes presented to a data server
> no longer authorized to perform them will be honored.  Consider a 
> situation in which a client's layout is recalled and either a write 
> from that client is delayed in transit, or the client mistakenly 
> writes to the old data server. It is important for server implementations 
> to properly protect against such write requests. Note that even though 
> data transport protocols such as NFSv4.1 and iSCSI have features to 
> protect against delayed writes, these features are not likely to deal 
> with this particular issue since the delayed write may not be a replay 
> in the context of the connection to the old data server, and the write 
> which replaces and supersedes it is going to a different destination.

I don't understand the mention of iSCSI in this context, as if the
client waits for the iSCSI write completion, then a duplicate write
should not be possible.  There might be some error cases (but iSCSI's
error handling is quite good, complete with specified timeouts), or
problems resulting from misbehavior of a multi-pathing SCSI driver
above iSCSI (but even that is covered under the requirement to wait
for all writes to complete).

In light of Benny's comment, this entire paragraph (starting from
"For storage protocols ..." and including the text below) looks like
it's file-layout-specific and hence ought to be moved to an appropriate
file layout section and modified to be specific to the file layout.
In doing so, keep in mind that sessions are "MUST use" with respect
to pNFS, and check that the scenarios involving delay of duplicate
operations are still possible with sessions and the requirement
on clients to wait for write completions.

> In light of this issue, implementations need to take particular care to 
> ensure that writes are not executed when it is invalid to do so. The 
> exact means by which this is done will depend on details of the server 
> implementation. The implementation will have to reliably ensure that 
> either such write requests can never be received by the old data server, 
> that the data server is reliably informed that it must not act on such 
> write requests, or that the data server no longer has physical access to 
> the storage in question.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Wednesday, June 21, 2006 7:11 PM
> To: Goodson, Garth
> Cc: [email-masked]
> Subject: RE: [nfsv4] pnfs: lingering writes
> 
> 
> At the risk of upsetting this, at least the block/volume layout
> view of the "lingering writes" issue is roughly:
> - Trust the client not to mistaken (or maliciously) write, as
> 	enormous damage far beyond the layout is possible (e.g.,
> 	physical filesystem metadata may be exposed, allowing a
> 	truly malicious client could wipe out all the superblocks).
> - Client MUST wait for write completions before returning the
> 	layout that allows the writes.  This removes delay in
> 	transit as an issue.
> The only thing that's left is protocol replay, and that's dealt
> with by not using protocols that can cause replays.  The latter
> one ("Client MUST wait") ought to be a general pNFS requirement.
> 
> I think my primary discomfort with Garth's proposed text is that
> it's not clear that the "implementations need to take particular
> care to  ensure that writes are not executed when it is invalid
> to do so" statement seems to exclude the block/volume layout's
> reliance on the client to address significant portions of this
> issue ("executed" seems to imply that the client already sent
> the write).
> 
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
> 
> > -----Original Message-----
> > From: Garth Goodson [mailto:[email-masked]] 
> > Sent: Wednesday, June 21, 2006 6:50 PM
> > To: Noveck, Dave
> > Cc: Benny Halevy; Black, David; [email-masked]
> > Subject: Re: [nfsv4] pnfs: lingering writes
> > 
> > I slightly edited Dave's text.  I would like to submit this 
> > to Spencer 
> > soon, if I don't hear otherwise.
> > 
> > As well, I'm not sure where this text should be placed.  As Dave 
> > mentioned earlier it seems out of place in it's current section
> (Storage 
> > Device Recovery).  One possibility is in the 'Recalling a Layout' 
> > section -- comments?
> > 
> > Text:
> > 
> > Within pNFS, it is possible that write requests may be honored by a
> data 
> > server no longer allowed to perform them.  This can occur after a 
> > client's layout is recalled and either a write from that client is 
> > delayed in transit, or the client mistakenly (or maliciously) writes
> to 
> > the old data server. It is important for server implementations to 
> > properly protect against these write requests. Note that 
> even though 
> > data transport protocols such as NFSv4.1 and iSCSI have features to 
> > protect against delayed writes, these features are not 
> likely to deal 
> > with this particular issue since the delayed write may not 
> be a replay
> 
> > in the context of the connection to the old data server, 
> and the write
> 
> > which replaces and supersedes it is going to a different 
> destination.
> > In light of this issue, implementations need to take particular care
> to 
> > ensure that writes are not executed when it is invalid to 
> do so. The 
> > exact means by which this is done will depend on details of 
> the server
> 
> > implementation. The implementation will have to reliably 
> ensure that 
> > either such write requests can never be received by the old data
> server, 
> > that the data server is reliably informed that it must not 
> act on such
> 
> > write requests, or that the data server no longer has 
> physical access
> to 
> > the storage in question.
> > 
> > -Garth
> > 
> > 
> > Noveck, Dave wrote:
> > > How about this:
> > > 
> > > Because pnfs can, through the recall of layouts, invalidate 
> > access via
> > > one data server, and require use of the meta-data server 
> or possibly
> > > another data server, it is important for implementations 
> to properly
> > > protect against the possibility that write requests will 
> be honored
> > > by a data server no longer allowed to perform them.  In 
> situtations
> > > in which it is possible for a write to be delayed in transit, the
> > > possibility exists that a write will be received by a server no
> > > longer allowed to act on it.  Note that even though data transport
> > > protocols such as NFSv4.1 and iSCSI have features to 
> protect against
> > > lingering writes, these features are not likely to deal with this
> > > particular issue since the delayed write may not be a replay in 
> > > the context of the connection to the old data server, and 
> the write
> > > which replaces and supersedes it is going to a different 
> > destination.
> > > In light of this issue, server implementations need to take 
> > particular
> > > care to ensure that such delayed writes are not acted on 
> when it is 
> > > not valid to do so.  The exact means by which this is done 
> > will depend
> > > on details of the data server and meta-data server implementation,
> > > but the implementation will have to reliably ensure that 
> either such
> > > write requests cannot never be received by the old data 
> > server, that 
> > > the  data server is reliably informed that it must not 
> act on such 
> > > write requests, or that the data server no longer has 
> > physical access 
> > > to the storage in question.  
> > > 								
> > 				
> > >    
> > > 
> > > -----Original Message-----
> > > From: Goodson, Garth 
> > > Sent: Thursday, June 15, 2006 5:43 PM
> > > To: Noveck, Dave
> > > Cc: Benny Halevy; 'David Black' (E-mail); [email-masked]
> > > Subject: Re: [nfsv4] pnfs: lingering writes
> > > 
> > > 
> > > 
> > > 
> > > Noveck, Dave wrote:
> > >> Now that I look at it, I'm very doubtful about this 
> > paragraph.  Rather
> > >> than augment it, I think we should just remove it.
> > >>
> > >> First of all it is not clear to me why "this approach", 
> > i.e. rewriting
> > >> data that you once wrote asynchronously, to where it 
> > supposed to go,
> > >> after you find out you can't write it to the original place makes
> > >> you any more subject to a lingering write problem (if your 
> > transport
> > >> protocol has one to begin with).  If your protocol has a 
> > lingering write
> > >> problem then you have a lingering write problem; get a 
> > better protocol,
> > >> but nowhere do we propose using a protocol with such a 
> problem as a
> > >> transport protocol.
> > >>
> > >> Also the first two paragraphs don't talk about situations 
> > subject to the
> > >> lingering write problem.  They talk about a situation is 
> > which you have 
> > >> had async writes acknowledged and async write only apply to file
> > >> variant.
> > >>
> > >> But even if you had the lingering write problem, i.e. you 
> > did a write,
> > >> got no response, and then wrote to the MDS or a 
> different DS, then 
> > >> sessions isn't needed to save you.  What saves you is that 
> > the storage
> > >> management protocol should have told the old DS to ignore 
> > writes done
> > >> under the old layout and if he is not and going to let the old DS
> > >> write data now held by the new DS then he belongs in the 
> > Broken pNFS
> > >> Implementation Hall of Fame (or Shame)
> > > 
> > > I think this is the key.  Perhaps we should replace it with 
> > something 
> > > that gets this point across.
> > > 
> > > -Garth
> > > 
> > > 
> > > 
> > > , but there isn't anything that
> > >> sessions can do about it, since the lingering write may be 
> > the first 
> > >> instance of that write request that he did.
> > >>
> > >> I guess there is a case for somewhere saying "The 
> > surgeon-general has
> > >> determined that using transport protocols subject to the 
> lingering 
> > >> write problem can cause you grief" so that someone 
> inclined to use
> > >> v2/v3/v4.0 as a transport protocol is appropriately 
> warned off, but
> > >> I don't think it makes sense in this section.
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Benny Halevy [mailto:[email-masked]] 
> > >> Sent: Thursday, June 15, 2006 10:09 AM
> > >> To: Goodson, Garth; 'David Black' (E-mail)
> > >> Cc: [email-masked]
> > >> Subject: [nfsv4] pnfs: lingering writes
> > >>
> > >>
> > >> Garth, the last paragraph in section 14.7.4 talks about 
> > the "lingering
> > >> write" 
> > >> problem. Its last sentence says "However, this is solved 
> > by sessions in
> > >> NFSv4.x"
> > >> I'd like to add to the paragraph that the problem is 
> also solved by
> > >> iSCSI error 
> > >> recovery as of RFC3720.  This is applicable to OSD over 
> iSCSI and I
> > >> believe also 
> > >> to SCSI block commands over iSCSI.
> > >>
> > >> Benny
> > >>
> > >>
> > >> _______________________________________________
> > >> nfsv4 mailing list
> > >> [email-masked]
> > >> https://www1.ietf.org/mailman/listinfo/nfsv4
> > 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Tue, 27 Jun 2006 12:26:34 -0700

Ok.  So it sounds as if we want a small paragraph, probably associated 
with the layout return section that is general, sets the ground rules, 
and specifies that each layout type needs to specify its semantics.

We can then move the majority of the section into the file-layout 
section.  I do want to make sure that this issue is addressed in both 
the block and object drafts.

-Garth

---

From: Garth Goodson <[email-masked]>
Date: Fri, 30 Jun 2006 14:09:37 -0700

I've taken people's feedback and generated this paragraph.  Even with 
the rules outlined by David Black, lingering writes may still exist if 
the metadata server is forced to unilaterally revoke the layout given 
the client's untimeliness in returning a recalled layout.

Within pNFS, it is possible that write requests may be presented to a 
data server no longer allowed to perform them.  pNFS limits this
behavior by requiring that a client MUST wait for completion of all
writes covered by a layout range before returning a layout that covers 
that range.  Since, the server has no control as to when the client will 
return the layout the server may, at some time, decide to unilaterally 
revoke the client's access provided by the layout in question.  Upon 
doing so the server must deal with the possibility of lingering writes, 
outstanding writes still in flight to data servers identified by the 
revoked layout.  Each layout-specification MUST define whether 
unilateral layout revocation by the metadata server is supported, and if 
so, the specification must also outline how lingering writes are to be 
dealt with; e.g., data servers identified by the revoked layout in 
question could be fenced off from the appropriate client.  If unilateral 
revocation is not supported, the specification must ensure there is no 
possibility of outstanding write requests when a layout is returned.

-Garth

Garth Goodson wrote:
>>> writes to the old data server. It is important for server 
>>> implementations to properly protect against such write requests. Note 
>>> that even though data transport protocols such as NFSv4.1 and iSCSI 
>>> have features to protect against delayed writes, these features are 
>>> not likely to deal with this particular issue since the delayed write 
>>> may not be a replay in the context of the connection to the old data 
>>> server, and the write which replaces and supersedes it is going to a 
>>> different destination.
>>
>>
>>> to ensure that writes are not executed when it is invalid to do so. 
>>> The exact means by which this is done will depend on details of the 
>>> server implementation. The implementation will have to reliably 
>>> ensure that either such write requests can never be received by the 
>>> old data server, that the data server is reliably informed that it 
>>> must not act on such write requests, or that the data server no 
>>> longer has physical access to the storage in question.
>>
>>
>> Thanks,
>> --David
>>
>>
>>>> From: Garth Goodson [mailto:[email-masked]] Sent: 
>>>> Wednesday, June 21, 2006 6:50 PM
>>>> To: Noveck, Dave
>>>> I slightly edited Dave's text.  I would like to submit this to 
>>>> Spencer soon, if I don't hear otherwise.
>>>>
>>>>> particular issue since the delayed write may not be a replay in the 
>>>>> context of the connection to the old data server, and 
>>>
>>> the write
>>>
>>>>> From: Goodson, Garth Sent: Thursday, June 15, 2006 5:43 PM
>>>>> To: Noveck, Dave
>>>>>> From: Benny Halevy [mailto:[email-masked]] Sent: Thursday, 
>>>>>> June 15, 2006 10:09 AM
>>>>>> write" problem. Its last sentence says "However, this is solved 
>>>>
>>>> by sessions in
>>>>
>>>>>> NFSv4.x"
>>>>>> iSCSI error recovery as of RFC3720.  This is applicable to OSD over 
>>>
>>> iSCSI and I
>>>
>>>>>> believe also to SCSI block commands over iSCSI.
>>>>>>
>>>>>> Benny
>>>>>>
>>>>>>