Subject: [nfsv4] parallel opens in NFSv4.1 ?

From: eric kustarz <[email-masked]>
Date: Fri, 11 Aug 2006 12:13:58 -0700

Not sure if this has been clarified or not yet... but based on a 
prototype, we found that allowing parallel opens for an open owner can 
be a big performance win for NFSv4:
http://nasconf.com/pres05/kustarz.pdf

In NFSv4.0, there was this text that basically states all requests for 
an open owner should be serialized:
"
8.1.5. Sequencing of Lock Requests

Locking is different than most NFS operations as it requires "at- 
most-one" semantics that are not provided by ONCRPC. ONCRPC over a 
reliable transport is not sufficient because a sequence of locking 
requests may span multiple TCP connections. In the face of 
retransmission or reordering, lock or unlock requests must have a well 
defined and consistent behavior. To accomplish this, each lock request 
contains a sequence number that is a consecutively increasing integer. 
Different lock_owners have different sequences. The server maintains the 
last sequence number (L) received and the response that was returned. 
The first request issued for any given lock_owner is issued with a 
sequence number of zero.

Note that for requests that contain a sequence number, for each 
lock_owner, there should be no more than one outstanding request.
"

I believe this has been cleaned up in the minorversion in section 7.1.5 
to only apply to lock owners (so not to opens).

Now that sessions replaces the seqid logic for opens, does NFSv4.1 allow 
us to send parallel opens for an open owner?  Or is there something else 
that needs to be added/removed from the protocol to allow this?  Has 
anyone with sessions prototypes experimented with this?

eric

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 11 Aug 2006 15:26:07 -0400

> an open owner should be serialized:

I don't understand why you can't solve this problem in 4.0 by just
creating more openowners as needed on the fly.

Lockowners have an additional meaning (they are also used to determine
when there is a locking conflict) which prevents you from creating new
ones arbitrarily.  I didn't think there was a similar problem with open
owners.  Am I missing something?

--b.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 11 Aug 2006 15:40:02 -0400

> anyone with sessions prototypes experimented with this?

If you send 2 open requests that with the same open_owner across
different slots, then how are you going to figure out which of the 2
unordered replies contains the correct server state?
As far as I can see, the only way would be to try a READ or WRITE, and
then see whether or not it returns NFS4ERR_OLD_STATEID. I suppose you
could do that, but how will that affect your numbers?

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 11 Aug 2006 16:26:39 -0500

> could do that, but how will that affect your numbers?

At one time, the stateid.seqid field was meant to allow the client to
determine the ordering of the stateids returned from a server and to
allow the client to determine which stateid is the most recent.
However, given the discussion we have had about stateid generation
at the server and the determination that the server does not need
to change the stateid on state changes, that is unreliable.

As has been pointed out, the lack of the _CONFIRM step allows the
client to create more open-owners; for the posix client, it is
likely not desireable to have an open-owner with greater granularity
than a process because of file locking issues (although it could
be done).  And don't forget that the extra state on the client
means extra state (memory footprint) at the server.

For the workloads Eric has in mind, it is sufficient to have 
open-owner/process granularity.

Spencer

---

From: eric kustarz <[email-masked]>
Date: Tue, 15 Aug 2006 10:22:30 -0700

>to change the stateid on state changes, that is unreliable.

I missed that part on the server changing stateids, but it seems like 
the restriction of:
"Note that for requests that contain a sequence number, for each 
lock_owner, there should be no more than one outstanding request."

could be changed to something like:
"There SHOULD be no more than one outstanding request for requests that 
are potentially modifying the same stateid."

The Solaris client creates an open owner from <mntinfo, cred> - thatis, 
no stateid information in the open owner.  We have the open stream data 
structure that is <open owner, file>, and that contains the stateid.

This would allow multiple open owner requests from the same <process, 
cred>, as long as the open was for a different file.

The impossible part of the parallel open prototype was that the client 
never knew the server's "seqid window" - that is, how many parallel 
open/seqid requests i could have in flight at the same time (much like 
the TCP window).  I thought sessions gave us this window.

As i understand it, the open owner has gone from an object containing a 
name and seqid to just containing a name.  Seems unnecessary to 
serialize on just a name.

eric

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 15 Aug 2006 14:23:45 -0400

> serialize on just a name.

The serialisation has nothing to do with the name. The serialisation
allows you to determine unambiguously which stateid is the current one
for a given <filehandle,open owner> pair for the case where you have
multiple OPENs on the wire.

If you can determine this stateid through other means, then you may not
need to serialise.

Cheers,
  Trond

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 15 Aug 2006 14:15:50 -0500

> If you can determine this stateid through other means, then you may not

It would likely ease the client implementation if it could count on the 
server returning a different stateid (seqid++) at each state changing
operation.  Is there interest in enforcing this stateid behavior so
the client's can have multiple OPENs in flight?

Spencer

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 15 Aug 2006 16:29:36 -0400

> the client's can have multiple OPENs in flight?

That would certainly resolve the ambiguity.

Cheers,
  Trond

---

From: eric kustarz <[email-masked]>
Date: Thu, 17 Aug 2006 16:45:22 -0700

>the client's can have multiple OPENs in flight?

So this is my take: sessions allows the client to know that it can send 
up to 'ca_maxrequests' outstanding requests (assuming they each fit 
within 'ca_maxrequestssize') and the server has enough resources to 
handle those requrests.  This means then the client can send up to 
'ca_maxrequests' opens at the same time for the same open owner.

This doesn't however guarantee the stateid handling is done correctly 
(as Trond was asking about).  A client can send multiple parallel opens 
on *unique* files and it should work fine.  I would guess this is bigger 
impediment to better performance and looks like sessions solves it.

Bumping the stateid's seqid, as Spencer mentions, would get us one step 
farther in that we could reliably send multiple parallel opens of the 
same open owner on any file (including the same one).

eric