Subject: RE: [nfsv4] pnfs: lingering writes

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Jun 2006 08:34:07 -0400

How about this:

Because pnfs can, through the recall of layouts, invalidate access via
one data server, and require use of the meta-data server or possibly
another data server, it is important for implementations to properly
protect against the possibility that write requests will be honored
by a data server no longer allowed to perform them.  In situtations
in which it is possible for a write to be delayed in transit, the
possibility exists that a write will be received by a server no
longer allowed to act on it.  Note that even though data transport
protocols such as NFSv4.1 and iSCSI have features to protect against
lingering writes, these features are not likely to deal with this
particular issue since the delayed write may not be a replay in 
the context of the connection to the old data server, and the write
which replaces and supersedes it is going to a different destination.
In light of this issue, server implementations need to take particular
care to ensure that such delayed writes are not acted on when it is 
not valid to do so.  The exact means by which this is done will depend
on details of the data server and meta-data server implementation,
but the implementation will have to reliably ensure that either such
write requests cannot never be received by the old data server, that 
the  data server is reliably informed that it must not act on such 
write requests, or that the data server no longer has physical access 
to the storage in question.  

-----Original Message-----
From: Goodson, Garth 
Sent: Thursday, June 15, 2006 5:43 PM
To: Noveck, Dave
Cc: Benny Halevy; 'David Black' (E-mail); [email-masked]
Subject: Re: [nfsv4] pnfs: lingering writes

Noveck, Dave wrote:
> Now that I look at it, I'm very doubtful about this paragraph.  Rather
> than augment it, I think we should just remove it.
> 
> First of all it is not clear to me why "this approach", i.e. rewriting
> data that you once wrote asynchronously, to where it supposed to go,
> after you find out you can't write it to the original place makes
> you any more subject to a lingering write problem (if your transport
> protocol has one to begin with).  If your protocol has a lingering write
> problem then you have a lingering write problem; get a better protocol,
> but nowhere do we propose using a protocol with such a problem as a
> transport protocol.
> 
> Also the first two paragraphs don't talk about situations subject to the
> lingering write problem.  They talk about a situation is which you have 
> had async writes acknowledged and async write only apply to file
> variant.
> 
> But even if you had the lingering write problem, i.e. you did a write,
> got no response, and then wrote to the MDS or a different DS, then 
> sessions isn't needed to save you.  What saves you is that the storage
> management protocol should have told the old DS to ignore writes done
> under the old layout and if he is not and going to let the old DS
> write data now held by the new DS then he belongs in the Broken pNFS
> Implementation Hall of Fame (or Shame)

I think this is the key.  Perhaps we should replace it with something 
that gets this point across.

-Garth

, but there isn't anything that
> sessions can do about it, since the lingering write may be the first 
> instance of that write request that he did.
> 
> I guess there is a case for somewhere saying "The surgeon-general has
> determined that using transport protocols subject to the lingering 
> write problem can cause you grief" so that someone inclined to use
> v2/v3/v4.0 as a transport protocol is appropriately warned off, but
> I don't think it makes sense in this section.
> 
> 
> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Thursday, June 15, 2006 10:09 AM
> To: Goodson, Garth; 'David Black' (E-mail)
> Cc: [email-masked]
> Subject: [nfsv4] pnfs: lingering writes
> 
> 
> Garth, the last paragraph in section 14.7.4 talks about the "lingering
> write" 
> problem. Its last sentence says "However, this is solved by sessions in
> NFSv4.x"
> I'd like to add to the paragraph that the problem is also solved by
> iSCSI error 
> recovery as of RFC3720.  This is applicable to OSD over iSCSI and I
> believe also 
> to SCSI block commands over iSCSI.
> 
> Benny
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Wed, 21 Jun 2006 15:49:43 -0700

I slightly edited Dave's text.  I would like to submit this to Spencer 
soon, if I don't hear otherwise.

As well, I'm not sure where this text should be placed.  As Dave 
mentioned earlier it seems out of place in it's current section (Storage 
Device Recovery).  One possibility is in the 'Recalling a Layout' 
section -- comments?

Text:

Within pNFS, it is possible that write requests may be honored by a data 
server no longer allowed to perform them.  This can occur after a 
client's layout is recalled and either a write from that client is 
delayed in transit, or the client mistakenly (or maliciously) writes to 
the old data server. It is important for server implementations to 
properly protect against these write requests. Note that even though 
data transport protocols such as NFSv4.1 and iSCSI have features to 
protect against delayed writes, these features are not likely to deal 
with this particular issue since the delayed write may not be a replay 
in the context of the connection to the old data server, and the write 
which replaces and supersedes it is going to a different destination.
In light of this issue, implementations need to take particular care to 
ensure that writes are not executed when it is invalid to do so. The 
exact means by which this is done will depend on details of the server 
implementation. The implementation will have to reliably ensure that 
either such write requests can never be received by the old data server, 
that the data server is reliably informed that it must not act on such 
write requests, or that the data server no longer has physical access to 
the storage in question.

-Garth

---

From: [email-masked]
Date: Wed, 21 Jun 2006 19:11:07 -0400

At the risk of upsetting this, at least the block/volume layout
view of the "lingering writes" issue is roughly:
- Trust the client not to mistaken (or maliciously) write, as
	enormous damage far beyond the layout is possible (e.g.,
	physical filesystem metadata may be exposed, allowing a
	truly malicious client could wipe out all the superblocks).
- Client MUST wait for write completions before returning the
	layout that allows the writes.  This removes delay in
	transit as an issue.
The only thing that's left is protocol replay, and that's dealt
with by not using protocols that can cause replays.  The latter
one ("Client MUST wait") ought to be a general pNFS requirement.

I think my primary discomfort with Garth's proposed text is that
it's not clear that the "implementations need to take particular
care to  ensure that writes are not executed when it is invalid
to do so" statement seems to exclude the block/volume layout's
reliance on the client to address significant portions of this
issue ("executed" seems to imply that the client already sent
the write).

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> From: Garth Goodson [mailto:[email-masked]] 
> Sent: Wednesday, June 21, 2006 6:50 PM
> To: Noveck, Dave
> Cc: Benny Halevy; Black, David; [email-masked]

> write requests, or that the data server no longer has physical access
to