Subject: [nfsv4] Different interpretations of stateid.

From: [email-masked]
Date: Wed, 21 Jun 2006 17:26:38 -0400 (EDT)

> So I'm wondering what various interpretations of stateid's people have?

Ah, a favourite topic of mine. I suggest starting by a careful read of
Sec. 14.2.2, particularily IMPLEMENTATION:
   Even though CLOSE returns a stateid, this stateid is not useful to
   the client and should be treated as deprecated.  CLOSE "shuts down"
   the state associated with all OPENs for the file by a single
   open_owner.  As noted above, CLOSE will either release all file
   locking state or return an error.  Therefore, the stateid returned by
   CLOSE is not useful for operations that follow.

Since the Close is for a file and the Stateid becomes "not useful" after
that, it seems to me that an open_stateid is associated with a file.
(My interpretation is that an OpenStateid is associated with the file
 opened and the open_owner used to open the file.)

A lock_stateid seems to me to be a slightly different beast. It seems to
represent all byte range locks for a given LockOwner, but does it have
to be specific to a file or a particular Open of a file? This one is far
less obvious, but I'd argue that the critical statement is also in the
description of Close:
   If record locks are held, the client SHOULD release all locks before
   issuing a CLOSE.  The server MAY free all outstanding locks on CLOSE
   but some servers may not support the CLOSE of a file that still has
   record locks held.  The server MUST return failure if any locks would
   exist after the CLOSE.

This implies that the server must know which byte range locks are associated
with which Open being closed. But if a client has done multiple Opens of
the same file (with different open_owners) how does it know which Open the
locks are associated with?

To me, the only way this association is established is via a Lock Op with
the open_to_lock_owner argument and this returns a lock_stateid.

So, to me a lock_stateid is used for all byte range locks for a given
lockowner associated with a given Open of the file. (This implies that
there can be multiple stateids for the same lock_owner. It also implies that
it is associated with a file, since the Open is associated with a file.)

Since this is not spelled out in RFC3530, I tried to code my server in
such a way that it would still work for clients that use the same
lock_stateid for all byte range locks done by the same lock_owner, even
if they don't seem to be "associated" to the same Open, as above.

rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 23 Jun 2006 17:34:46 -0400

> My interpretation was always that the stateid is associated with the
file.

So's mine, but more to the point so is Rick's.  You appear to be in
violent
agreement.

> I think you took "not useful for operations that follow" in 14.2.2 out
of 
> context.  Here's the full context:

Clearly you can always give more context but "out of context" suggests
that Rick somehow shortchanged us by not giving more context.  I don't
have that impression.

> My quote would then be "CLOSE "shuts down" the state associated with
all OPENs 
> for the file by a single open_owner".  Hence the stateid represents
"the state
> associated with all OPENs for the file by a single open_owner" and
"operations
> that follow" relate to state changing operations issued by this
open_owner
> on this file.

To me "not useful for operations that follow" means that there are no
operations
for which it useful.  Clearly, for every stateid, there are some
operations
for which it is not useful.

I conclude that Rick is right and that this statement does at least
suggest that
stateids are associated with files and are not simply a shorthand for an
owner.
I also note that if the relationship between stateid and owner were
intended to
be a simple one-to-one relationship, the spec would be expected to say
so.  You 
can't make a conclusive case from the absence of something, but it does
add 
weight to the view that the each stateid is limited to a file and is not
just 
a shorthand for an owner.

Also I think Rick's second point is important.  Even if a stateid is not
tied
to a file, a lock stateid cannot just be a shorthand for a lockowner but
has to
include some notion of the openowner, so that it would at least have to
encode
an openowner/lockowner pair even if it didn't include an indication of
the 
file.

Rather than engage in a detailed picking apart of the spec for the "true
meaning"
of stateid however, I think we should focus on making sure that existing
clients
and servers can be upgraded to v4.1.  We do not want to make a choice
here
that will invalidate the basic data structure handling of any client or
server
no matter how arguably correct that would be based on an analysis of the
intent 
of RFC3530.  After all, the authors of RFC3530 disagree on what stateid
means
(which does suggest a problem in our review process that I'd like not to
repeat).

So given that many servers exist which give out different stateids for
different
files, even with the same owner, it seems clear that no client can be
written
depending on an assumption that the stateid is a shorthand for an owner,
That
is, it cannot be surprised or object to getting two different stateids
for the
same openowner (which it might if it were a shorthand for an openowner)
and
it can't use one stateid instead of another it got when opening a
different
file (which would be valid if a stateid were a shorthand for an
openowner,
but will not work on such servers).

So it seems clients, even their authors believe that the stateid is a
shorthand
for an owner, must act as if it is tied to a file, or at least they must
be
extraordinarily tolerant of servers that act as if a stateid is tied to
a file.

So this leaves a question about servers.  Are there servers out there
that act
on the assumption that a stateid does not include an indication of the
file
and that an open stateid is a shorthand for a openowner, and that a lock
stateid
is a shorthand for lockowner/openowner pair (I don't think it would work
for it
to be a shorthand for a lockowner)?  I think whether any such servers
exists is
very important in deciding how we address this going forward.

If such servers exist then I think we have to define the clients view so
that
it accommodates either approach.  We still might want to do this even if
such
servers don't exist, but we don't have to.

One way that might resolve this is the following:

     At a conceptual level, a stateid designates a file/openowner pair
(for an
     open stateid) or a file/openowner/lockowner triple.

     The stateid assignment spaces from the client viewpoint are to be
treated
     as segmented by filehandle, so that any comparison of the bit
patterns of
     stateids associated with different filehandles or a transfer of one
to the
     other is prohibited.

     A server may choose to assign the same bit pattern to stateids
associated
     with different filehandles so that the same owner is associated
with the
     same bit pattern on different files but if it does not it must not
allow
     a filehandle to be used with an inappropriate stateid.  For
example, if 
     I open A with owner Q and get stateid X.  I may not use that
stateid with
     file B if owner Q never opened file B.  That needs to get a
BAD_STATEID
     error.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, June 22, 2006 2:08 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Different interpretations of stateid.

[email-masked] wrote:
> In case my post yesterday wasn't clear (I'm usually not:-), here's an
> example assuming that an open_owner isn't tied to a file:
> - client opens foo with owner "xxx", gets stateidXXX
> - client opens bar with owner "xxx", gets stateidXXX
> - client closes foo with stateidXXX
> *** stateidXXX is "not useful for operations that follow"
> - client closes bar with stateidXXX
> --> server fails this with something like NFS4ERR_BADSTATEID, since
stateidXXX
>     was rendered "not useful for operations that follow" by the
previous close
> 
> Of course "not useful for operations that follow" is quoted from the
end of
> the IMPLEMENTATION para of sec. 14.2.2.
> 
> Whereas, if the server issues distinct stateids for each open:
> - client opens foo with owner "xxx", gets stateidXXXfoo
> - client opens bar with owner "xxx", gets stateidXXXbar
> - client closes foo with stateidXXXfoo
> - client closes bar with stateidXXXbar
> 
> The only way I can see the first example working is if "not useful for
> operations that follow" doesn't mean what I read it to mean? rick

My interpretation was always that the stateid is associated with the
file.
I think you took "not useful for operations that follow" in 14.2.2 out
of 
context.  Here's the full context:

    locking state or return an error.  Therefore, the stateid returned
by

My quote would then be "CLOSE "shuts down" the state associated with all
OPENs 
for the file by a single open_owner".  Hence the stateid represents "the
state
associated with all OPENs for the file by a single open_owner" and
"operations
that follow" relate to state changing operations issued by this
open_owner
on this file.

---

From: [email-masked]
Date: Sat, 24 Jun 2006 19:35:16 -0400 (EDT)

> I'm a little lost--has anyone identified an actual problem here?  I don't think
> I've yet seen anyone given an example where a client and a server would fail to
> interoperate because their implementors took two different "interpretations" of
> the stateid.

Ah, Bruce throws down the gauntlet. I can't resist...

First, I'd argue that my example in the previous email would apply, since the
client would get NFS4ERR_BADSTATEID or some such error, for the second Close.
Also, how about the following...
Suppose there is a server that assumes a lock stateid is a shorthand for
a lock owner and a client that is Posix-like and maps each process to
an owner name.

- P1 opens foo with open_owner "P1"
- server replies with OpenStateIdP1
- P1 forks child P2, which inherits open on foo (we'll call this fromp1_fdesc)
- P2 write locks byte 0 of foo with OpenStateIdP1 and lock owner "P2"
  (using open_to_lockowner case, for fromp1_fdesc)
- server replies with LockStateIdP2
- P2 opens foo with open_owner "P2" (we'll call this one p2_fdesc)
- P2 write locks byte 1 of foo with OpenStateIdP2 and lock owner "P2"
  (using open_to_lock_owner case, for p2_fdesc, since it assumes that this
   establishes what the server considers "outstanding locks" for Close and
   wants to close OpenStateIdP2 separately from OpenStateIdP1)
*** at this point, it is not obvious to me what the server would do, but
    it seems it would either return LockStateIdP2 again or return an error
    (lets assume it returns LockStateIdP2 again)
- server replies with LockStateIdP2
- P2 unlocks byte 1 of foo with LockStateIdP2
- P2 Closes foo using OpenStateIdP2
- server replies NFS4ERR_LOCKED, since byte 0 is still locked for foo and
  lock owner "P2"
Form the Posix-like client's perspective, the lock on byte 0 is for
fromp1_fdesc and does not need to be unlocked before the close on p2_fdesc.

rick

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 25 Jun 2006 08:00:16 -0400

I think any time the spec has statements that contradict each other,
then that's a problem.  It may not be an enormous problem, but if
we are doing a substantial revision of the spec and we are at least 
doing one, for v4.1 and might do another (an rfc3530bis), we should
come up with language that helps people understand the protocol, and
does not contradict itself. 

> I'm a little lost--has anyone identified an actual problem here?  
> I don't think I've yet seen anyone given an example where a client 
> and a server would fail to interoperate because their implementors 
> took two different "interpretations" of the stateid.

If either of these two statement were taken seriously by a client,
you would have a problem with a server which took a different approach.

If a stateid is a shorthand for an owner, then a stateid returned for
two different opens with the same openowner will be the same and can
be used interchangeably.  That would work with a server that took the
same approach but wouldn't work with the Netapp server of the Solaris
server.

On the other hand, if a client assumed that a stateid designated a 
file-owner pair and that each such designation was therefore unique
within a client, it might organize a client-wide structure to match
sateid to file-owner pairs.  I don't know why it might do that but
it would be justified from one sort of explanation of stateid.  That
would work with the Netapp server, but might fail with others.

Now neither of those has happened and I think that is because people
have essentially ignored the explanation and used the details of 
open/close/lock, etc. and not gone much beyond them.  In this case, 
that is a good thing, but I think we should have explanatory text
that people don't have to ignore to get things that work.

> And, in fact, it's not the job of the protocol specification to say 
> what different objects "mean".  Its job is to describe how compliant 
> implementations use them. 

I think that's true with regard to such objects as the owner.  We 
really can't say much about what it means.  We handwave a bit and
can't get much beyond that because it relates to how protocol objects
relate to worl beyond.

With regard to stateid, the situation is different, it is not the 
meaning of stateid as an object in the real world that we are concerned
with but the relationship between protocol objects, and for that I
think the protocol should have a clear position on that and explain
it.  Thus we need the implementor to understand that an owner is 
within the context of a client and not useful outside it, and that
a stateid either has a one-to-one relationship with an owner or it
doesn't.

> If we can assign them a simple consistent "meaning", that may be 
> an aid in understanding or implementing the spec.  

It is also an aid to producing a spec that doesn't have problems.
It appears that we have dodged a bullet here and gotten by despite 
the fact there is this difference of opinion but generally speaking,
if the authors of the spec don't agree on what an object is, there
is a good chance that the protocol will have problems.

> But the essential thing is to make sure that the description of 
> how they are to be used is complete and correct.

I agree that that is essential, but I also feel that we can't publish
a spec that we know contradicts itself.  Given that we have to delete
the contradictory stuff, we could replace it with "A stateid is a 
128-bit tag returned by and required by various operations." and 
leave it at that, but it seems kind of unfriendly.  I'm going to 
try to come up with something that is more helpful.

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Saturday, June 24, 2006 6:39 PM
To: Noveck, Dave
Cc: Benny Halevy; [email-masked]; [email-masked]
Subject: Re: [nfsv4] Different interpretations of stateid.

On Sat, Jun 24, 2006 at 05:12:04PM -0400, Noveck, Dave wrote:
> So we have two different explanations of stateid in the spec, they 
> contradict each other, and further they are contradicted in turn by 
> various pieces of the details of the protocol defined by the spec.
> We seem to have a bit of a mess.

I'm a little lost--has anyone identified an actual problem here?  I
don't think I've yet seen anyone given an example where a client and a
server would fail to interoperate because their implementors took two
different "interpretations" of the stateid.

> On the other hand, if you just use the stateid provided by open, lock,

> etc., in the obvious ways suggested by the operation definitions and 
> ignore the explanatory text it seems that we have a protocol that 
> works despite the fact that different people have different 
> interpretations of what a stateid really is

And, in fact, it's not the job of the protocol specification to say what
different objects "mean".  Its job is to describe how compliant
implementations use them.

If we can assign them a simple consistent "meaning", that may be an aid
in understanding or implementing the spec.  But the essential thing is
to make sure that the description of how they are to be used is complete
and correct.

--b.