Subject: RE: [nfsv4] pNFS - file commit

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 5 Oct 2005 11:01:07 -0400

> While not having thought about it very deeply, it seems that 
> allowing the client to change the target for a data write may
> open a series of either data inconsistencies or build assumptions
> into the meta-data and data server interactions.  Is this desired?

I've been digging into this a little bit more deeply as a result
of your mail.  There's some interesting stratigraphy here and so
far at least, I haven't hit a toxic waste dump.  I don't see the 
inconsistencies that you are worried about.

The other problem is that if you dig at the alternative site in
which you can't switch, you do hit some material you'd rather not
have to deal with.

Let's try following the approach that says that you can't switch
the target and see where it takes us.

I write a 4K block of data to a DS, asynchronously, and get a 
response.  Now the layout is recalled.  If I am unable to switch
the target, then I must, at this time, COMMIT the write successfully,
i.e. if the verifier fails I must write it again and makes sure it
is securely written to disk, and I must do this completely before
returning the layout.  And if the DS is unresponsive the layout
cannot be returned.

Why is this so?  If I were to return the layout and do the COMMIT
later, then I would have no way of dealing with a verifier mismatch.
I certainly could not redo the write to the DS when I had no layout.
That may no longer be the proper location of the data.  By recalling
the layout the MDS has revoked my right to do IO to the DS.  So if
I can't switch to writing the data at a new valid location, MDS or
DS according to new layout, my updates are lost.  This would be 
very bad.

So, if we are not allowed to switch, then we have to succeed with the
write-commit before returning the layout.  But this would be disastrous
since it means that if the server failed or I could not reach it for 
some reason, I could not return the layout.  And if it were the case
that the failure of a server like this could prevent a layout from 
being returned indefinitely, then MDS's would have to be very tolerant
of failures to return layouts, since they might not be the client's
fault.

Consider this scenario.  Client one has problems communicating with 
DS-A, so the MDS is informed (do we have a way to do this?) and now
moves the data on DS-A to DS-B and recalls the layout so that people
can know about the change.  Now client two, on the same subnet as
client one, runs into the same communication issues and can't get
a response to his COMMIT and he never will until the communication
problem is resolved.  The result is that the switch to DS-B cannot
be effective until you decide to shoot client two, who is just an
innocent bystander.  Again, very bad.

> For example, if the data server may not be responding to the NFSv4
> client because the data server has a network partition between
> itself and some other component of the NFSv4(pNFS) server.  If the
> client switches to the meta-data server for data writes after
> having instantiated a WRITE to the data server, then the modified
> data at the data server may still be "in-flight" because of its
> own network partition.

I'm not sure what you mean by in-flight here.  In the case we are 
talking about the async writes have been acknowledged and are not
what I would consider "in-flight".  The other case, in which you
have done a write (sync or async) and haven't got a reply does 
raise some similar issues and I will deal with that below.

> As mentioned, the client may see its own
> data over-written by the initial WRITE to the data server.  

Not in this case, since my initial WRITE was done and replied-to
and I haven't done any others.  But even if I did do more such 
writes and were interleaving them, we would have a case of me 
overwriting my data with the same bytes that I wrote.  I don't 
see any data inconsistency.

> If
> that is to be protected against, then there is an assumption
> that the meta-data server has control in such a way to determine
> that modified data received by the data server in question is
> stale in some fashion.

I think there is a lingering-write issue that does pose data 
inconsistency issues, but it is not related to async write.

Suppose I do WRITE, and I get no response.  While I'm breaking my
connection, trying to re-establish a new one, I get a recall of 
the layout.  Requiring me to wait until the server comes up again 
has problems similar to those we have discussed in the async case.
On the other hand, if return the layout and then do the IO to the
MDS there is an issue.  It is not that there is conflict between
the DS IO and the MDS IO.  They are two writes of the same data.
The issue is that after the MDS IO is replied-to, I go on to do 
N more WRITE's and then seventeen WRITE's later the DS write 
happens.  That's not good.

So it appears that we have a lingering write problem in this case.

> Obviously this has not been a problem in the standard NFSv4 world
> since there is only one funnel in which to stuff data but this
> ability to redirect seems complicated.  Unnecessarily so?  I am not
sure. 

Not so obvious.  Let's look at an analogous case without pNFS.  I
do a write and get no response.  I break the connection, or at least 
I break my side of it and wait a while and then establish a new one
and send my write.  If I get an reply to that, then I proceed with N
writes, and if somehow the original write (stuck in a router somewhere)
is now acted on, I have the same lingering write problem that I just
mentioned in the pNFS case.  It doesn't seem to me that it has anything
to do with the fact that I have pNFS or that I switched to the MDS.  The
problem is that I have a write whose status is uncertain, that I
replace it by another write (whether to the same server or a different
one) and treat the successful completion of that write as allowing me
to proceed, not worrying about the possibility of the lingering write,
whose status remains uncertain.

If we move beyond standard v4 to standard v4.1 (i.e. with sessions),
we have a solution to that problem, in that each write whether 
reissued on a new connection or not can only be executed once (EOS).
Once I get a reply for my retried write, I know that the initial
write can no longer be executed.

So in this context, allowing me to switch to writing to the MDS
does have an issue and that is that by writing in a different session,
I no longer have the EOS protection that I would by having my
write be a retry recognizable as such by the sessions logic.

Given that it seems like we have three choices (one not so well-
defined yet):

     Not allow the switch to the MDS when you have an outstanding
     replied-to write (done async write with reply is OK).  Has 
     some pretty negative effects for error recovery.  

     Allow the switch, imposing a delay to try to be at least as good
     as v4.0 in preventing a lingering write.  The good point is that
     it is no worse than v4.0.  The bad point is that it is no better.

     Provided some sort of mechanism to allow the MDS to prevent
     the lingering write.  Details are not clear but I think it should
     be oriented toward preventing the effects of the lingering
operation
     rather than on denying access to a particular set of data.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]]
Sent: Monday, October 03, 2005 6:31 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS - file commit

While not having thought about it very deeply, it seems that 
allowing the client to change the target for a data write may
open a series of either data inconsistencies or build assumptions
into the meta-data and data server interactions.  Is this desired?

For example, if the data server may not be responding to the NFSv4
client because the data server has a network partition between
itself and some other component of the NFSv4(pNFS) server.  If the
client switches to the meta-data server for data writes after
having instantiated a WRITE to the data server, then the modified
data at the data server may still be "in-flight" because of its
own network partition.  As mentioned, the client may see its own
data over-written by the initial WRITE to the data server.  If
that is to be protected against, then there is an assumption
that the meta-data server has control in such a way to determine
that modified data received by the data server in question is
stale in some fashion.

Obviously this has not been a problem in the standard NFSv4 world
since there is only one funnel in which to stuff data but this
ability to redirect seems complicated.  Unnecessarily so?  I am not
sure.

Spencer

On Mon, Noveck, Dave wrote:
> > Machines/devices can disappear and be 
> > replaced by otters and we should have more elaborate recovery than
just 
> > wait for them to come back.
> 
> We should have more functional recovery than just waiting for them to
> come back, but I hope it would not be very elaborate.
> 
> When we talk about the async write case, there a couple of important
> points to note:
> 
>     The only one who has the information necessary to recover is the
>     client since he has the dirty data and probably nobody else does.
> 
>     If the client redoes his async writes when he many not have to,
>     this is only a possible performance issue so that any solution 
>     would tend to err on the side of writing these a little bit more
>     than necessary.
> 
> So I would think that what the spec should say is something like:
> 
> When writes to a data server are async and thus require a COMMIT to
> be done to that data server, the client is not obligated to wait 
> indefinitely for that data server to become available again in order
> to do the COMMIT.  The client, rather than hold the asynchronously
> written data indefinitely is encouraged to and can make sure that
> the data is written by using other paths to that data.  The client 
> may write the data to the metadata server, either synchronously or
> asynchronously with a subsequent COMMIT.  Once it does this, there 
> is no need to wait for the original data server.  In the event that
> the data range to be committed is transferred to a different data 
> server, as indicated in a new layout, the client may write to
> that data server and COMMIT there if necessary, and should consider
> the transfer of responsibility for the data to the new server as
> strong evidence that that is more efficacious way to get the data 
> written.  In either case, once the COMMIT (or synchronous WRITE) is 
> validly done (to either data server or metadata server), there is no 
> need to continue attempting to COMMIT to the original data server 
> or wait for it to become available.  That data server may never be 
> visible to the client again. 
> 
> -----Original Message-----
> From: Marc Eshel [mailto:[email-masked]] 
> Sent: Monday, October 03, 2005 4:20 PM
> To: Trond Myklebust
> Cc: [email-masked]; [email-masked]
> Subject: Re: [nfsv4] pNFS - file commit
> 
> 
> [email-masked] wrote on 10/03/2005 12:43:42 PM:
> 
> > m? den 03.10.2005 Klokka 14:49 (-0400) skreiv Jim Zelenka:
> > 
> > > I agree that COMMIT to the MDS should result in appropriate
COMMITs
> > > to the data storage devices, just like a regular NFSv4 write to
the 
> MDS
> > > should result in the correct set of writes being directed to the
> > > data servers. I don't think that we should mandate that COMMIT
always
> > > go through the MDS, however.
> > 
> > Once again. I'd very much like to see a proper discussion/proposal
on
> > how to implement this by the people advocating it.
> > 
> First let me say that I don't think the reboot verifier that was use
with 
> a single server is an appropriate solution for the more complex system

> that we will have with pNFS. Machines/devices can disappear and be 
> replaced by otters and we should have more elaborate recovery than
just 
> wait for them to come back. We now have the MDS that can help in the
case 
> of a DS failure. One way to do it is discover at LAYOUTCOMMIT that one
or 
> more DSs are down. It can than provide a different layout or request
that 
> all writes be directed to the MDS. I know this not a complete proposal
we 
> all should think some more about this multi device environment
recovery 
> than just assume that they will just reappear. I know for example that
if 
> in a cluster file system environment a machine will crash I will
recall 
> all layout that included that machine and try to replace it with a 
> different machine or work without it.
> Marc.
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Fri, 07 Oct 2005 14:56:04 -0700

Noveck, Dave wrote:
>      operation rather than on denying access to a particular set of data.
> 

After talking with Dave, I have a suggestion on option three.

The biggest problem seems to occur if the write is delayed within the DS 
to the point that the client retries the write elsewhere.  At that point 
it becomes an issue of either killing the pending write, wherever it may 
be queued, or making sure that the write has no effect.  The former is 
fairly complicated given the system implementation.  The latter, can be 
done by invalidating all state associated with the file, as long as the 
entity responsible for performing the write validates the write, e.g., 
by checking for a valid stateid.  After all state has been invalidated, 
you know that either the write succeeded or was killed off.  Note, this 
approach also works if the write is queued externally to the DS.

However, it would be bad, from a client's perspective, if suddenly all 
state associated with a file disappeared from a DS.  I think we should 
think about an operation that a client can perform at the MDS that hands 
back a new stateid that ensures all previous ops (from that client for 
that file) are fenced off from that point on.  The internal 
implementation can invalidate the DS state, and restablish new state 
such that the client can retry its write through the metadata server 
without worrying about lingering effects; since older, lingering writes 
will be rejected due to the out-of-date stateid it carries.

As Dave commented, I think it is important that the client is able trade 
stateid's with no (visible) modification of the underlying state they 
represent.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Fri, 7 Oct 2005 20:59:51 -0700

> will be rejected due to the out-of-date stateid it carries.
If the MDS can communicate with the DS in order to invalidate the stateid 
why can't it just tell the DS to stop accepting IO for the given stateid. 
If the MDS can not communicate with the DS than we just replaced a stateid 
and the only DS that needs this information doesn't. What is the propose 
of replacing stateid? We should redirect the IO to the MDS or another DS 
and do whatever is possible on the back-end to block future IO through 
that DS. That might include telling the DS to stop accepting IO if you can 
communicate with it, or zone it out, and so on, this is a back-end 
problem. What we need in the protocol is a way for the client and MDS to 
agree that there is a problem with a particular DS. This information can 
go from the client to the MDS or from the MDS to the client. After this 
exchange the client will redirect its IO and the MDS will do its best to 
disable IO to that DS or at least the IO for the given stateid. 
Marc.