Subject: [nfsv4] DESTROY_SESSION and clientid trunking

From: <[email-masked]>
Date: Mon, 19 Jul 2010 17:22:01 -0400

This mail concerns the use of DESTROY_SESSION in the context of clientid
trunking.  Specifically, there are going to be cases in which a client
will encounter a situation in which connections to one session are
unresponsive, whether due to client-side networking issues, a wire being
cut, server-side networking issues, or a node of a clustered server
being down.  No matter what the cause the client will need to transfer
work from one session to another.  For simplicity, let's assume we have
clientid CL and with two sessions S1 and S2 and that S1 becomes
unresponsive making it necessary to transfer work so that everything for
the client is being done on a single session S2.

The problem I'd like to consider is that of modifying-idempotent
requests such as WRITE.  The question of non-idempotent requests such as
RENAME is harder and probably requires some work in v4.2, but the
simpler case is important since WRITEs are typically much more common.

So the issue is that if you have a request issued on S1 and we'll use
WRITE throughout as the example, and you get no response, then you would
like to issue the WRITE on the other session.  Since WRITEs are, in the
strict sense, idempotent, you don't have a problem if two WRITEs are
issued but what is a problem is if there were a WRITE issued on S1, and
it was just lazily hanging around somewhere, and then if you did the
WRITE on S2, and it completed then if the WRITE on S1 were to spring
back to life, you have the likelihood of data corruption.

What you need is the assurance that the first WRITE (the one on S1),
either succeed or failed but is not sitting around waiting to be done at
a later point.  You want to draw a line under all requests initiated as
part of S1.  I'm assuming the right way to do that is to destroy S1.
Does anybody have any other ways to do that?  Things like closing the
file will assure that the writes are done but they have unsatisfactory
locking issues in that someone can open the file with a deny-mode and
you might not be allowed to open it again.

However, there are some problems in the way that DESTROY_SESSION is
specified and I think they need to be addressed, whether as errata in
the v4.1 context or in the v4.2 context.

The first problem is that you would expect to have such a guarantee
specified in the definition of DESTROY_SESSION and it isn't there.  You
could argue that since it specifies that the reply information is
dropped, it would be bad for requests to terminate in a situation where
there is session reply information to update, but that isn't ironclad.
I think it makes sense for the initial paragraph to be changed to
something like this:

   The DESTROY_SESSION operation closes the session and discards the
   session's reply cache, if any.  All pending operations initiated on 
   the session are terminated and no additional ones can be started.
   Thus, the requester is assured, once it receives the response, that
   no operations initiated on the session will modify any file data 
   or attributes or locking state associated with the client.

   Any remaining connections associated with the session are immediately

   disassociated.  If the connection has no remaining associated
sessions, 
   the connection MAY be closed by the server.  Locks, delegations,
layouts, 
   wants, and the lease, which are all tied to the client ID, are not 
   affected by DESTROY_SESSION.

What do people think?  Is this reasonable?  Is there some way we can get
by without it?

The other issue is that the spec says:

   DESTROY_SESSION MUST be invoked on a connection that is associated
   with the session being destroyed.

So here I have a couple of questions:
*	Is there a good reason for this?  Does it make sense?
*	Is this a big problem for the issue we are talking about?

I believe the answers to both are "no", but I'm not sure.

The problem I'm worried about is that if you are doing a DESTROY_SESSION
to clean up from a non-responsive session, if you have to issue it on
connections associated with that session.  So the problem would be that
the DESTROY_SESSION can only be sent when you don't need it to be sent,
i.e. when the destination IP's are functioning, but if they are, you
don't want to send it.

There may be security issues but the aren't the security issues dealt
with by the text that follows the text above.  Would there really be any
security issue allowing this on any connection that is associated with a
session that is part of the same clientid as the session being
destroyed?  After all, you can do an EXCHANGE_ID with a new client
verifier and get rid of all of all sessions as long as you can confirm
it my creating one new session.  Given that you can doing that why can't
you destroy the session.

So one possible way of getting around this would be to associate
connections that are part of S2 (let's call them with C2A and C2B) with
S1 just so they would be allowed in the context in which you need them.
While the text talks about the security issues of making sure that we
have the right server, it doesn't talk about how you might reject the
BIND_CONN_TO_SESSION because it is for a connection that is for the
wrong so_minor_id.  Is the server obliged to reject this?  If it isn't
then it might have a connection associated with a different session,
e.g. C2A with S1.  If it allows that, is the server obliged to accept S1
requests (in general) on the same basis as it accepts S2 requests?  Or
can it treat these specially, i.e. slow for normal requests (because
they are routed over a cluster interconnect to the node normally
handling S2), but capable of using clustering knowledge to respond to
DESTROY_SESSION even when the S1 node is not functioning?

Is there anything in the spec that prevents this?

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 19 Jul 2010 17:42:18 -0400

> the client is being done on a single session S2.

I don't get it--isn't it the *connections* associated with the session
that become unresponsive, not the session itself?

If you wanted EOS across sessions, then it seems to me that you'll end
up creating another session abstraction on top of sessions.

And just fixing whatever bug is causing the session "unresponsiveness"
would seem simpler than the infinite regress....  But I probably don't
understand what you mean: how could a session become unresponsive?

--b.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Mon, 19 Jul 2010 21:06:45 -0400

> that become unresponsive, not the session itself?

OK, but if the single connection associated with a session or all the
connections associated with a session becomes unresponsive, then you
still have the same issue whether you are saying the session is
unresponsive or the connections.

I think I neglected to mention one important piece of context.  If you
have two sessions associated with a single clientid and this is the
result of the client on its own deciding on doing this form of clientid
trunking, then what you say is true.  

However, if the two sessions are established because the server returned
different so_minor_id values, then we have a different situation.  In
this case the server is telling you that two different IP's are not
capable of supporting a single session.  If that is the case, typically
it is because you have a clustered server such that the two IP's
terminate in different hardware which is incapable of sharing memory.
With shared memory, you would simply be able to support a common session
and its associated replay cache.

In such cases, it is very likely that a session or rather the set of all
sessions sharing a given so_minor_id, will become unresponsive, as the
result of various hardware problems in addition to bugs that you can
simply fix. 

> up creating another session abstraction on top of sessions.

That may seem to be a flaw in terms of abstraction management, but it is
in response to a practical requirement.  Clients want an EOS guarantee
for their requests.  It is simple (by which I mean we have already done
it) to provide it within the context of a single session.  However, if
one is forced to move requests from one session to another, and if one
can be forced into either doing this or having the requests wait
forever, then if one is not willing to accept the possibility of data
corruption, one has to address the issue of how to make sure the old
request cannot come to life.

Note that this doesn't fully address the issue of EOS across sessions
for non-idempotent requests and I think some v4.2 changes would be
required, but v4.1 does allow different so_minor_id value for clustered
servers, and it seems kind of rough to say you can only do this if you
accept the possibility of data corruption.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Monday, July 19, 2010 5:42 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] DESTROY_SESSION and clientid trunking

--b.

---

From: Tom Haynes <[email-masked]>
Date: Tue, 20 Jul 2010 10:24:07 -0500

> accept the possibility of data corruption.

Without a clear picture of what v4.2 means, is this the type of thing we 
want to put in it?

I'm not asking if it is something we want to fix, but rather is v4.2 the 
right vehicle?

If I look at v4.2, I see it as a collection of small optional feature 
sets - pathless objects,
hole punching, space reservation, server to server copy, dedup, 
metastripe, access
check, etc. I could be wrong, but that seems to be what people are 
proposing.

I don't see it as fixing things we got subtly wrong in either 4.0 or 4.1.

Perhaps even releases introduce new functionality and odd releases are 
informational?
Even ones driven by design and odd ones driven by implementation experience?

And that raises another question, according to your slide deck:
http://www.ietf.org/proceedings/10mar/slides/nfsv4-9.pdf

We don't want another huge document. But at what point do we want a 
comprehensive
document that rolls up all of the smaller v4.x changes from v4.1? I.e., 
when does it make
sense to have one document to consult versus 10 other ones?

---

From: <[email-masked]>
Date: Tue, 20 Jul 2010 14:00:25 -0400

> Without a clear picture of what v4.2 means, is this the type of thing
we 

Without a clear picture of what v4.2 means, how could I answer that? :-)

Clearly, I've made a mistake.  Let me change:

    and I think some v4.2 changes would be required

To

    and I think some changes in v4.2 or a subsequent minor version 
    would be required

There seems to be an inconsistency between saying we couldn't do this
because we don't have a clear picture of what v4.2 means, and then also
saying we couldn't do this because we do have a clear picture and this
doesn't fit.  Right now it doesn't fit because there is no substantive
proposal to address it.  I was just saying I didn't see how that issue
could be addressed in v4.1 but I could be wrong about that.

As to the things that have been proposed, I think your characterization
is accurate, but I don't see that it should be controlling.  If we had a
large change that was inconsistent with dealing other things that were
in in a reasonable time, that would be a reason to exclude it, but if
you have a bunch of small features adding ops, then putting in an op to
correct an issue we found doesn't seem so discordant.  After all, Bruce
might argue that this is the "cross-session request EOS" feature and he
would not necessarily be wrong.  For a lot of things, the line between
correction and feature isn't so clear. 

> I don't see it as fixing things we got subtly wrong in either 4.0 or
4.1. 

If not, would we have a whole minor version devoted to fixing things we
got subtly wrong in earlier minor versions?  Fixing such things is
something that people recognize should be done but it just won't hold
the group's interest for an entire minor version (and that of
implementers and of executives allocating their time to work on clients
and servers).  It has to be combined with other things and the simplest
model is to do them as soon as someone has a clear solution that people
agree makes things better.  That may not be v4.2 for this issue but if
someone does have a reasonable way of addressing it, I don't see why it
should be excluded.

> sense to have one document to consult versus 10 other ones?

The first answer is "not for v4.2" and I haven't heard anybody disagree
about that.

But you are right that if v4.x gets a lot higher this will be an issue.

I think it is clear we will never have a document bigger than 650 pages.

We'll need, at that point, not one document but a set of documents, not
organized by version but organized by topic.

At some point there will be a need for RFC5661bis and I think we should
make a big step in that direction, at that point.  So we might have,

    An introductory document containing basically chapters 1-4, 14, 16,
17, 19, 21, 22.
    A file attributes document containing chapters 5-6.
    A namespace document containing chapters 7 and 11.
    A locking document containing chapters 8-10.
    A pNFS document containing chapters 12 and 13.

Those document would be written so that covered all versions (except
v4.0).  Most of them would not have to be updated for a minor version.
You could then periodically write a new document that covered the
material in chapters 15, 18, 20, as updated for the current minor
version.  Alternatively, that document could be smaller, only containing
chapter 15 and an index of where to go for the current handling of each
op and callback.  In terms of using a web version of the document, there
wouldn't be much difference between a link within a document and one
between documents.  As long as you go to the right place, the fact that
many documents are referenced should not matter.  

The big issue with doing like this is that it is a lot of editing work,
so I think the x in v4.x will have get fairly high before people decide
they have to take the issue on.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Tom Haynes
Sent: Tuesday, July 20, 2010 11:24 AM
Cc: [email-masked]
Subject: [nfsv4] What does v4.2 look like?

want to put in it?

right vehicle?

I don't see it as fixing things we got subtly wrong in either 4.0 or
4.1.

Perhaps even releases introduce new functionality and odd releases are 
informational?
Even ones driven by design and odd ones driven by implementation
experience?

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 20 Jul 2010 18:10:23 -0400

> terminate in different hardware which is incapable of sharing memory.

If an entire server has become unresponsive, then you're really doing
failover--at which point, don't you need a persistent and/or distributed
reply cache if you really want exactly once semantics?

--b.

---

From: <[email-masked]>
Date: Mon, 26 Jul 2010 18:02:59 -0400

> reply cache if you really want exactly once semantics? 

First, we need to clarify the "entire server" which may be becoming
unresponsive.  Do we mean the set of things having the same value of
so_major_id and so_minor_id, which I'm going to call a server-minor?  Or
do we mean the set of things having the same value of so_major_id and
any set of potentially different so_minor_id values, which I'll call a
server-major.

If the entire server which has become unresponsive is a server-major,
then yes you have failover.  So the client will typically do an
EXCHANGE_ID to establish a clientid and depending on whether there are
persistent sessions he will either have the EOS that he really needs or
not.  But what is important in terms of the issue we are talking about
here is that the EXCHANGE_ID draws a line under everything that happens
under the previous clientid.  Whether you have EOS or not, what the
client cannot live with is a situation which a WRITE started under the
previous clientid might happen later, after I start doing IO's under the
new clientid.  That is a recipe for data corruption.

Now consider the same situation for the case in which a server-minor is
what becomes unresponsive.  You are not going to establish a new
clientid but if you want to issue a WRITE previously issued on a
different server-minor, you need an analogous guarantee, that everything
previously issued to that server-minor has either been done or it hasn't
and never will.  What you can't live with a situation in which it might
happen at any subsequent time. 

Leaving aside the case of read-only data, everybody needs true EOS,
whether they are attentive enough to want it or not.  But v4.1 does not
make true EOS with persistent sessions a mandatory feature, probably
because of implementation difficulty.  

Now in our example of an unresponsive server-minor, if there is support
to update the session reply cache on disk or on another backup piece of
hardware, then we have EOS in this case.  It doesn't show up as
persistent sessions but rather as connections becoming unresponsive and
when they are re-established, they will wind up on an alternate piece of
hardware with the correct reply cache being available, whether from disk
or otherwise.

But just as we deal with servers that don't have persistent sessions in
the server-major case, we should deal as well as we can, when we have
the case of a server-minor which does not have the ability to store its
reply cache information persistently.  I agree that people need EOS for
this case, but we do have to deal with the case in which we don't have
the needed infrastructure.  In the case, I'm thinking about a case in
which we have unresponsive connections to a server-minor and we need to
be able assure that operations previously started on these are either
done or not and what is critical is that they are not done in the
future.  

You could do this explicitly  with an op that take a server-owner or you
could to this at the session level.  It seems to me that doing that as
part of DESTROY_SESSION makes sense, but you would have to deal with the
current spec issues that make this difficult or impossible.  My feeling
is DESTROY_SESSION needs to assure people that ops started under that
session are not going to execute after the DESTROY_SESSION completes,
and I would like to understand if there is a good reason for the current
restriction as to what connection the op can be issued on.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Tuesday, July 20, 2010 6:10 PM
To: Noveck, David
Cc: [email-masked]
Subject: Re: [nfsv4] DESTROY_SESSION and clientid trunking

--b.

> > requests such as WRITE.  The question of non-idempotent requests
such
> as
> > either succeed or failed but is not sitting around waiting to be
done
> at
> > security issue allowing this on any connection that is associated
with
> a
> > e.g. C2A with S1.  If it allows that, is the server obliged to
accept
> S1