Subject: [nfsv4] proposal to make stateid4.seqid transparent

From: Spencer Shepler <[email-masked]>
Date: Tue, 22 Aug 2006 13:39:29 -0500

In the following I make the case that in NFSv4.1, the use of the
stateid4.seqid can provide a reduction in state managed by the client
while increasing the level of parallelism and therefore performance as
observed by the NFSv4.1 client.

In draft-05, it is stated that the stateid field, stateid4.seqid, is
no longer needed and can be used by the server as it sees fit; it is
opaque to the client.  While there is no issue with correctness in
this, a transparent, monotonically increasing stateid4.seqid can
assist the NFSv4.1 client in providing better performance and
scalability.  The server would increment stateid4.seqid on each
operation such that the client could sort the responses and subsequent
use of the stateid4 would act, in a sense, as a membar for subsequent
operations.

As we all know, upon open, the NFSv3 client sends a LOOKUP/GETATTR to
the server (and in many cases just a GETATTR).  There is no
serialization requirements between file opens at the client (other
than the self imposed) and it is allowed for "full" parallelism.  This
is not the case for NFSv4.

With the structure of NFSv4.0, OPEN/CLOSE require the association to
the open-owner and its corresponding seqid.  The seqid, in this
context, provides for replay protection but it also provides for the
ordering of the OPEN/CLOSE operations for a particular open-owner.
Therefore, an NFSv4.0 client must choose between a broad association
of open-owner to client processes (the more processes included, the
less degree of parallelism but fewer OPEN_CONFIRM steps) or a narrow
association of open-owner to client processes (higher degree of
parallelism but more OPEN_CONFIRMs).  

There is also the scoping issue of the open-owner's seqid in NFSv4.0.
Once a open-owner is associated with a file as a result of OPEN,
subsequent operations using the stateid still require that the
open-owner's seqid be used.  This means that a CLOSE for a
unassociated file must be serialized with other OPENs; multiple CLOSEs
from the same client must be serialized if associated with the same
open-owner even if the CLOSEs are for different files across
filesystems.  This issue remains even if open-owners are fine-grained

There is an optimization available to the NFSv4.0 client but it is not
very desirable (error recovery path at the client is cumbersome).  The
client can send multiple operations for a single open-owner and hope
that the server executes them in the order specified by the seqid.  If
the server does, then great; if not, the client must recover from the
BAD_SEQID errors but resubmitted a request.

For NFSv4.1, we have removed the need for the open-owner/seqid pairing
with the use of the session sequence mechanism.  Note the sessions
mechanism doesn't provide a sequencing of operations outside of a
single slot (and hence one request at a time).  The other change in
NFSv4.1 is the recent clarification of how the stateid4.seqid may be
used; it is now opaque to the client.  And, the server is not required
to change the stateid4 on subsequent operations (OPEN upgrade) (have
we decided on this?)  Because of these, the client is now required to
serialize all requests related to the open-owner and the stateids
associated with the open-owner.

Certainly, the client implementor can now choose an association of
open-owner to local process that is more granular and achieve a higher
degree of parallelism as compared to NFSv4.0; there is no longer the cost
of OPEN_CONFIRM.  Even with this advantage, there is still the burden
of additional state (more open-owners) and the serialization as
required today in NFSv4.0 (in fact that serialization is a MUST).

So...

Even though the open-owner seqid is no longer used, for appropriate
processing, the server must serialize access to its data structures
for state bearing operations.  Therefore, it is possible to have
stateid4.seqid be transparent to the client and if the server were
required to monotonically increase stateid4.seqid value at each OPEN,
OPEN_DOWNGRADE, CLOSE, the client can benefit greatly.

With a transparent stateid4.seqid, the NFSv4.1 client can send OPEN,
OPEN(upgrade), CLOSE requests in parallel to the server.  This means
that a traditional NFS client could have one open-owner per principal
for all filesystems from the server.  The server would just execute
the operations as needed and return the execution ordering implied by
stateid4.seqid.  The client would then sort the responses based on
this seqid.  If a seqid "skips", then the client knows that an
intervening request was processed and would wait for the response to
arrive.  For example:

1)    OPEN(oo-A, file-A, READ)	      ->	  sid-1
2)    OPEN(oo-A, file-A, READ|WRITE)  ->	  sid-2
3)    CLOSE(sid-2)		      ->	  sid-3

In this case, the client knows that the OPEN in 2) was done before the
close and therefore the file is not "open".  If the two OPENs were
driven by two processes such that 2) should not be closed, the client
can resend the request.  Note this may happen if the client takes
advantage of the protocol's OPEN(dir, name).  This way the client
doesn't need to waste a round-trip on a LOOKUP(dir, name).  In so
doing, the client does open itself to this type of race but it should
be rare and the code to check for this condition should be reasonable.

There is a possibility that the client sends the two OPENs but the
second delayed at the server for some reason and the client is able to
CLOSE the first open.  The client can also detect this.  The stateid
results would be something like this:

1)    OPEN(oo-A, file-A, READ)	      ->	  sid-1
2)    OPEN(oo-A, file-A, READ|WRITE)  ->	  sid-3
3)    CLOSE(sid-1)		      ->	  sid-2

The client having knowledge of the OPEN ordering is helpful if it
needs to downgrade one of the OPENs.  For example:

1)    OPEN(oo-A, file-A, READ)	      ->	  sid-1
2)    OPEN(oo-A, file-A, READ|WRITE)  ->	  sid-2
3)    OPEN_DOWNGRADE(sid-2, READ)     ->	  sid-3

Therefore, I propose that in NFSv4.1 the stateid4.seqid be transparent
and monotonically increasing.

Spencer

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 22 Aug 2006 18:21:13 -0500

> observed by the NFSv4.1 client.

I have opened issue 116 for this.

Spencer

---

From: Mike Eisler <[email-masked]>
Date: Tue, 22 Aug 2006 16:40:50 -0700 (PDT)

> 2)    OPEN(oo-A, file-A, READ|WRITE)  ->	  sid-2

A couple points:

- NFSv4.1 adds OPEN by filehandle. Clients that need parallel
  OPENs should be looking at serializing the LOOKUP of the
  the final component and then sending parallel OPENs of the 
  file handle. Even without OPEN by filehandle, provisional
  rnodes indexed by directory filehandle and component name
  are possible, and such rnodes could be the point of
  synchronization/parallization.

- It seems to me the server has to serialize OPEN (or any other
  state changing operation), at least on
  a per open-owner/file basis, in order to safely bump stateid4.seqid.
  Otherwise, if it just bumps stateid4.seqid when it receives
  the operation or finishes the operation, 
  nothing prevents the server from finishing
  the parallel operations in a different order from when they were
  received. So the stateid4.seqid results are not meaningful.
  So this is either moving the serialization problem
  to the server (where it is harder to implement) or it isn't
  actually solving the problem. Taking one of Spencer's
  examples:

  Let's say the second OPEN and the CLOSE arrive more or less
  coincidentally. The second OPEN is processed. Before it
  acquires the mutex to bump stateid.seqid from sid-1 to
  sid-2, the CLOSE is processed (thereby closing the open state
  created by both OPENs). The CLOSE increments the stateid.seqid
  to sid-2. Now the other thread processing the second open
  gets the mutex, and increments stateid.seqid to sid-3.

  Perhaps the server has a read-shared/write exclusive mutex that
  allows OPENs to proceed in parallel, but CLOSEs and 
  OPEN_DOWNGRADEs serialize with each other and OPENs. 
  But all this can be done on the client. 

  Fully serializing operations on the server is problematic as
  it pins a thread or in event driven servers, an event context
  while the thread or context waits for the other threads or
  contexts to finish.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 22 Aug 2006 16:49:33 -0700 (PDT)

I realize there was one point in my note that I didn't
fully explain. Look for ^^^^^^^^ in the below

--- Mike Eisler <[email-masked]> wrote:

A couple points:

  The client thinks the CLOSE happened before the first
  OPEN, because sid-2 < sid-3. But in fact the server has
  closed the file.
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^