Subject: RE: [nfsv4] Preview of new locking chapter

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 2 Aug 2006 15:53:50 -0400

> Overall, I believe that the contents of the chapter need to be split 
> into three areas of discussion.

> The first would be the protocol's state and how it is instantiated, 
> manipulated, and destroyed/released.

> This would be in normal operation.  I would like to refer to this as 
> "state" instead of locks specifically to avoid any mis-interpretations

> of locks and byte-range locks.

I don't think you can talk about how it is instantiated and manipulated
because that is different for, byte-range lock, delegations, etc.  For
example owners apply to byte-range locks and opens, which is mostly what
the final section is about.

I think you can talk some of the general state management framework: the
concept of a stateid, leases, etc.
 
> Following this would be a discussion of the failure/recovery 
> mechanisms and their impact on the state (much of what is there in the

> chapter already.

What is there already is pretty much tied to byte-range locks and opens.
I think it would need some work to make it sufficiently general.

> Finally, a discussion of shares and byte-range locks.

But what about byte-range locks and shares?

Do you talk about how they are instantiated, manipulated, or is that in
the previous section?

> The reason for this is that these three areas are intermixed because 
> of the structure we had in rfc3530 and was never cleaned up.

I think they are still intermixed but it would certainly be good if we
could figure out how to clearly separate them.  I need to think about
how this can be done.

> The first is the issue of having the principal used at OPEN be the
> same one that is used for later use of the resultant stateid. Simply, 
> does the same user have to use the stateid on READ/WRITE as did the 
> OPEN.

There seemed to be a general agreement that we want to go in the
direction of tying credentials to owner state but there wasn't
sufficient discussion to determine exactly how far in this direction we
want to go.  There are a lot of issues here about the interaction of
state-owners and stateids.

We can start with your statement about credentials that must be used
when a stateid returned from open is used by READ, WRITE and presumably
truncate.

Then there is a related question about whether open-owners are tied to
credentials.  Can the same owner do two different opens with different
credentials?  

What about LOCK requests?  Must they use the same credential?  If not,
can READs and WRITEs using the locking stateid, use the lock credential
or the open credential?  Similarly can the same lock-owner use different
credentials on different open files?

What about using delegation stateids?

I think someone has to come up with a proposal in this area.

> The second item was a proposal that Bruce had made with respect
> to "blocking" lock requests.  Did we decide to pursue that?

I suspect that if you don't know that we have, then we haven't. That
doesn't mean we shouldn't though.

> Is there a scalability issue, that we care about, with the
> introduction of TEST_STATEID and FREE_BADLOCK?  If the client 
> receives the error indication on the SEQUENCE op results, how 
> does it go about clearing the condition?  At the moment, it 
> seems that the client must wonder through all of its state using 
> TEST_STATEID until the error in SEQUENCE goes away.  I suppose 
> that client's could essentially ignore that condition until it 
> received and error on the revoked state and then did the FREE_BADLOCK.

I think many clients will follow that approach, but I think we have to
have reasonable facilities for those that do want to get an immediate
indication of revoked locks.  Right now the V4.0 spec says 
you are supposed to do this, even when timers show you might possibly be
subject to a lease expiration.

So I think that what exists is reasonably scalable, because a client can
package a large number of TESTLOCKs in a single COMPOUND and can issue
these COMPOUNDs in parallel.  He will then have a list of revoked locks
and as each COMPOUND finishes he can do a FREE_BADLOCK on any that show
up as revoked.  I think that is reasonable but if it people don't think
it is, we could implement an op to get all the revoked stateids for a
client.  Then you need a cookie if the list is too big to come back and
then you have issues of how long such state 
is kept, what to do about changes in the list in the interim etc, 
so I'd like to stay away from that if we can.

> Also, RELEASE_LOCKOWNER should not be made mandatory not to implement.
> The LOCK request with the open_to_lock_owner4 argument served two 
> purposes.  The first was to establish the sequence number (which 
> we do not need with the use of sessions) but it also associates 
> the lock_owner with the open_owner.  

Indirectly.  Actually it associates the open-owner/lock-owner/file
triple with a stateid.  Once you have that stateid, you can release or
get additional locks by using stateid when you call LOCK.

> If RELEASE_LOCKOWNER is not available, the client will need to 
> deal with error returns for regular LOCK requests or make every 
> call with the open_to_lock_owner4 type. 

Certainly he doesn't have to do this if he has some lock for that
lock-owner/open-owner/file triple.

Whether he has to do it when he doesn't have a lock depends on the
lifetime rules for a stateid.  If your assumption is that a stateid
for a given lock-owner subordinate to given open can go away as 
soon as the last lock goes away (even when the open still exists),
then you would have to use open_to_lock_owner4, which doesn't seem
all that bad to me.

However, my assumption (not written down) is that the stateid lifetime
would not end until the open goes away.  If that were the case, then
you could continue to use the stateid until the file was closed.  
If you had a case in which a file was open forever and you continued
to use a large set of lockowner then perhaps there might be an issue
but I kind of feel that is unlikely.  If we did have this case, it
would simpler to give people a way to get rid of lock stateid's that
had no locks assigned to them.  Perhaps FREE_BADLOCK could be rename
so it would allowed to free any stateid for which there was actual 
associated valid lock?

> The RELEASE_LOCKOWNER operation should remain and used by the client 
> when it knows that the lockowner is no longer pursuing locks or makes 
> the determination itself (because of resource limits) that it can be 
> released.

We could put this back but it seems odd when the real issue is stateid
lifetime to control this at the lockowner level.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Wednesday, August 02, 2006 5:45 AM
To: [email-masked]
Subject: Re: [nfsv4] Preview of new locking chapter

On Sun, Noveck, Dave wrote:
> Here is the new revised-for-sessions locking chapter, including the
> changes for SEQUENCE-based notifications and a re-organized treatment 
> of lock revocations, that have been proposed previously.  Although the

> reaction to these was generally positive, there wasn't enough 
> discussion to really declare a consensus and so I promised everybody a

> chance to look at the new locking chapter, before it went into an
official draft.

After reviewing this chapter I have a few comments.

Overall, I believe that the contents of the chapter need to be split
into three areas of discussion.  The first would be the protocol's state
and how it is instantiated, manipulated, and destroyed/released. This
would be in normal operation.  I would like to refer to this as "state"
instead of locks specifically to avoid any mis-interpretations of locks
and byte-range locks.

Following this would be a discussion of the failure/recovery mechanisms
and their impact on the state (much of what is there in the chapter
already.

Finally, a discussion of shares and byte-range locks.

The reason for this is that these three areas are intermixed because of
the structure we had in rfc3530 and was never cleaned up.

A couple of comments that are not directly related to the specifics of
the locking chapter but were things I thought the WG had agreed to and
we need to confirm that and then find a home for them.

The first is the issue of having the principal used at OPEN be the same
one that is used for later use of the resultant stateid. Simply, does
the same user have to use the stateid on READ/WRITE as did the OPEN.

The second item was a proposal that Bruce had made with respect to
"blocking" lock requests.  Did we decide to pursue that?

For specific comments... 

Is there a scalability issue, that we care about, with the introduction
of TEST_STATEID and FREE_BADLOCK?  If the client receives the error
indication on the SEQUENCE op results, how does it go about clearing the
condition?  At the moment, it seems that the client must wonder through
all of its state using TEST_STATEID until the error in SEQUENCE goes
away.  I suppose that client's could essentially ignore that condition
until it received and error on the revoked state and then did the
FREE_BADLOCK.

Also, RELEASE_LOCKOWNER should not be made mandatory not to implement.
The LOCK request with the open_to_lock_owner4 argument served two
purposes.  The first was to establish the sequence number (which we do
not need with the use of sessions) but it also associates the lock_owner
with the open_owner.  If RELEASE_LOCKOWNER is not available, the client
will need to deal with error returns for regular LOCK requests or make
every call with the open_to_lock_owner4 type. The RELEASE_LOCKOWNER
operation should remain and used by the client when it knows that the
lockowner is no longer pursuing locks or makes the determination itself
(because of resource limits) that it can be released.

Spencer

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 2 Aug 2006 20:46:25 -0500

> could figure out how to clearly separate them.  I need to think about

Since this is more editorial in nature, let's take it off-list.

> when a stateid returned from open is used by READ, WRITE and presumably

Okay.

> credentials.  Can the same owner do two different opens with different

Yes. Especially improtant for clients that would like to minimize
the number of open-owners.

> credentials on different open files?

Again, for the client that minimizes the number of open-owners
the ability to have a different credential on LOCK would be good.

It seems that there needs to continue to be the stateid/credential
association to protect against the denial of service that is present.

> I think someone has to come up with a proposal in this area.

True.

> these COMPOUNDs in parallel.  He will then have a list of revoked locks

As long as we don't limit the size of COMPOUNDs too severly with
respect to the ERR_RESOURCE issue. :-)

> so I'd like to stay away from that if we can.

I agree.  I would prefer to avoid the enumeration feature as well.

I am okay with the idea that most clients will be happy to "ignore"
the SEQUENCE error until it stumbles into the stateid that is
causing the problem.  So maybe some text to that effect in the
discussion would be helpful.  I am also okay with the client
bundling TESTLOCK requests.  Just need to ensure that the 
TESTLOCK operation is defined such that it doesn't error out
so that a longer COMPOUND would have a chance to succeed.

One final nit is the names.  Can we have TEST_STATEID
and FREE_BAD_STATEID -or RELEASE_BAD_STATEID to generalize a litte?

> get additional locks by using stateid when you call LOCK.

Sorry for being imprecise, yes, it is the triple.

> then you would have to use open_to_lock_owner4, which doesn't seem

I was assuming the shortest duration of the lock-stateid guessing
that there would be an implementation that would make that choice.

My concern with the continual use of the open_to_lock_owner4
branch was the potential size of the lock_owner4.  Didn't seem
that we should punish the client becaused it chose a larger
value for lock_owner4.  Also thought it would simplify the
client implementation if it could count on the particular
encoding and not have to worry too much about an error coming
back from the server when it was in the middle of doing file
locking.

> but I kind of feel that is unlikely.  If we did have this case, it

I am not so sure.  We know that AIX uses a small number open-owners
and may continue to do so going forward (and other clients may
choose this path as well).  Seems prudent to account for such use.

> so it would allowed to free any stateid for which there was actual 

Did you mean to say that we rename FREE_BADLOCK to FREE_STATEID
and it would be used for the BAD stateid recovery as described
and add the ability to free/release a lock-stateid that was
associated with a valid open-stateid but did not have any
byte-range locks currently active?  Yes, I like that idea.

> lifetime to control this at the lockowner level.

Agreed.  Let's insert the ability to release the lock-stateid
from the server's bookkeeping.

Spencer