Subject: RE: [nfsv4] proposal to make stateid4.seqid transparent

From: [email-masked]
Date: Tue, 22 Aug 2006 18:05:28 -0400 (EDT)

> I agree with you about OLD_STATEID for v4.0 but I think parallel
> opens, enabled by sessions in v4.1, means that this does have an 
> important use, at least for CLOSE and OPEN_DOWNGRADE.
> 
> I agree though that this doesn't really provide any value in the case
> of LOCK, LOCKU, READ, WRITE, SETATTR, etc.  In those cases, the server
> can use the code part of the stateid to get what it thinks is the 
> current status of that stateid and act accordingly.  Rather than 
> depreate OLD_STATEID, I would simply remove that as a valid error
> for all the ops other than CLOSE and OPEN_DOWNGRADE.

Here's how I understand it:
- If a client issues multiple concurrent Opens on the same Open_Owner, it must
  wait until all the Opens reply, before allowing any of the StateIds returned
  from any of these Opens to be used (at least CLOSE, OPEN_DOWNGRADE).
- If it does this, then it can use the seqid field to sort out which stateid
  returned for Opens that happen to be for the same file (same FH in reply)
  to use.

This all makes sense to me and seems like a good idea.

Now what about returning NFS4ERR_OLD_STATEID for CLOSE, OPEN_DOWNGRADE?
I suppose this might be a useful client debugging tool, but the error only
occurs if the client has screwed up and not done the above. There are other
cases that will screw up with different errors if it allows CLOSE on one of
the Stateids before receiving all the Open replies.
For example, assume 2 concurrent Opens using the same Open_Owner on hard
links for the same file:
- The Opens are performed on the server in the order O1, O2, such that the
  Stateid in O2's reply in the current one.
- Client receives O2 reply and Closes it, which succeeds. (And also Closes
   O1, although the client doesn't realize that.)
- Client receives O1 reply and tries to use it. Gets NFS4ERR_BAD_STATEID?
> 
> As Trond has pointed out, for these ops, returning OLD_STATEID allows
> you to do parallel opens and closes even the presence of hard links,
> without any worries about things being done in the wrong order. 
I believe the above is a worry w.r.t. wrong order?

Also, what should a client do when it receives an NFS4ERR_OLD_STATEID reply?

A lazy client writer like me might just...
- CLOSE replies error. Hmm, I'm done with the state, so "tough shit", I
  don't care.:-)
- Then the Open state that doesn't get CLOSE'd is left dangling on the
  server.

I guess I have no problem with the server returning NFS4ERR_OLD_STATEID for
CLOSE, but I'm thinking that maybe it should discard the Open State at that
point anyhow, because the client might not know how to CLOSE it later.
(I was thinking that the server would not close off the state when it
 returned the NFS4ERR_OLD_STATEID reply.)

Make sense, or am I just "out to lunch"? rick

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 22 Aug 2006 18:31:10 -0400

>   from any of these Opens to be used (at least CLOSE, OPEN_DOWNGRADE).

If you are willing to do that, that would work.  I think that 
many people would find that degree of synchronization onerous 
for the .001% case in which hard links cause two opens which the
client thinks are independent to actually open the same file.

> - Client receives O1 reply and tries to use it. Gets
NFS4ERR_BAD_STATEID?

So it looks there are three possible answers to this case:

1) Your synchronization where we wait for all pending opens for a 
given lockowner before doing a close or a downgrade.  This still 
allows you do parallel opens but you may not do opens with closes.
If you have this synchronization, then I don't see the need for 
"transparent" seqids in stateid, but maybe others will disagree.

2) The client keys off seqid and only needs to synchronize when it 
sees a gap in the seqid sequence.  So in this case, if it sees O2
with a seqid gap for O1, it would wait at that point.  That requires
less synchronization, is more complicated, and would require
"transparent"
seqids.

3) Avoid the hard link problem by requiring server to assign separate 
stateid sequences for hard links and only merge/upgrade if both the
directory and the name match.  So in this case O1 and O2 in your example
would be independent on closing O2 would only close O2.  On the other
hand if the opens were for the same file (not hard links), the client
would have to wait for all pending opens before issue a close (or it
could simply single-thread opens and closes for the same file-owner
pair,
while file means the designated directory and filename rather than
the target filehandle).

-----Original Message-----
From: [email-masked]
[mailto:[email-masked]] 
Sent: Tuesday, August 22, 2006 6:05 PM
To: [email-masked]
Cc: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] proposal to make stateid4.seqid transparent

Here's how I understand it:
- If a client issues multiple concurrent Opens on the same Open_Owner,
it must
  wait until all the Opens reply, before allowing any of the StateIds
returned
  from any of these Opens to be used (at least CLOSE, OPEN_DOWNGRADE).
- If it does this, then it can use the seqid field to sort out which
stateid
  returned for Opens that happen to be for the same file (same FH in
reply)
  to use.

Now what about returning NFS4ERR_OLD_STATEID for CLOSE, OPEN_DOWNGRADE?
I suppose this might be a useful client debugging tool, but the error
only
occurs if the client has screwed up and not done the above. There are
other
cases that will screw up with different errors if it allows CLOSE on one
of
the Stateids before receiving all the Open replies.
For example, assume 2 concurrent Opens using the same Open_Owner on hard
links for the same file:
- The Opens are performed on the server in the order O1, O2, such that
the
  Stateid in O2's reply in the current one.
- Client receives O2 reply and Closes it, which succeeds. (And also
Closes
   O1, although the client doesn't realize that.)
- Client receives O1 reply and tries to use it. Gets
NFS4ERR_BAD_STATEID?

Also, what should a client do when it receives an NFS4ERR_OLD_STATEID
reply?

I guess I have no problem with the server returning NFS4ERR_OLD_STATEID
for
CLOSE, but I'm thinking that maybe it should discard the Open State at
that
point anyhow, because the client might not know how to CLOSE it later.
(I was thinking that the server would not close off the state when it
 returned the NFS4ERR_OLD_STATEID reply.)

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 22 Aug 2006 21:20:05 -0400

In the presence of share reservations, you have to have a lock
around the point at which you decide whether the open can be
granted (given other opens' denial modes) and you then
atomically (relative to any other open on that same file)
mark the file as upgraded as appopriate and add that open's
denial mode.  It is at that point that you would presumably
bump the sequence id.  You can't do it when you recieve the
operation or finish the operation, unless you want to do
the whole operation in a single-threaded fashion.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]]
Sent: Tuesday, August 22, 2006 7:41 PM
To: [email-masked]
Subject: Re: [nfsv4] proposal to make stateid4.seqid transparent

--- Spencer Shepler <[email-masked]> wrote:

> 1)    OPEN(oo-A, file-A, READ)	      ->	  sid-1
> 2)    OPEN(oo-A, file-A, READ|WRITE)  ->	  sid-2
> 3)    CLOSE(sid-2)		      ->	  sid-3

A couple points:

- NFSv4.1 adds OPEN by filehandle. Clients that need parallel
  OPENs should be looking at serializing the LOOKUP of the
  the final component and then sending parallel OPENs of the 
  file handle. Even without OPEN by filehandle, provisional
  rnodes indexed by directory filehandle and component name
  are possible, and such rnodes could be the point of
  synchronization/parallization.

- It seems to me the server has to serialize OPEN (or any other
  state changing operation), at least on
  a per open-owner/file basis, in order to safely bump stateid4.seqid.
  Otherwise, if it just bumps stateid4.seqid when it receives
  the operation or finishes the operation, 
  nothing prevents the server from finishing
  the parallel operations in a different order from when they were
  received. So the stateid4.seqid results are not meaningful.
  So this is either moving the serialization problem
  to the server (where it is harder to implement) or it isn't
  actually solving the problem. Taking one of Spencer's
  examples:

 1)    OPEN(oo-A, file-A, READ)	      ->	  sid-1
 2)    OPEN(oo-A, file-A, READ|WRITE)  ->	  sid-3
 3)    CLOSE(sid-1)		      ->	  sid-2

  Let's say the second OPEN and the CLOSE arrive more or less
  coincidentally. The second OPEN is processed. Before it
  acquires the mutex to bump stateid.seqid from sid-1 to
  sid-2, the CLOSE is processed (thereby closing the open state
  created by both OPENs). The CLOSE increments the stateid.seqid
  to sid-2. Now the other thread processing the second open
  gets the mutex, and increments stateid.seqid to sid-3.

  Perhaps the server has a read-shared/write exclusive mutex that
  allows OPENs to proceed in parallel, but CLOSEs and 
  OPEN_DOWNGRADEs serialize with each other and OPENs. 
  But all this can be done on the client. 

  Fully serializing operations on the server is problematic as
  it pins a thread or in event driven servers, an event context
  while the thread or context waits for the other threads or
  contexts to finish.

---

From: Peter Varga <[email-masked]>
Date: Tue, 22 Aug 2006 23:00:02 -0400

All this complexity seems to stem from having the restriction of only one
instance of an OPENed file per open-owner on the server. A CLOSE will close
all opens of the file, and hence the need for OPEN_DOWNGRADE.

Drawing from Dave's proposal #3, could someone please remind me why doesn't
OPEN just return separate stateids (stateid sequences) for every OPEN to a
file?

Thanks,
Peter

> From: Noveck, Dave [mailto:[email-masked]] 
> Sent: Tuesday, August 22, 2006 6:31 PM
> To: [email-masked]; [email-masked]
> Cc: [email-masked]