Subject: open upgrade/downgrade, NT and UNIX

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 17 Nov 2000 14:18:04 -0800

Here's my current take about what to do about the 
open upgrade/downgrade issue.  For background on this 
issue, see the following mail messages:

http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2000/0332.html
http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2000/0338.html
http://playground.sun.com/pub/nfsv4/nfsv4-wg-archive/2000/0353.html

This issue is difficult to resolve because:

      NT is open-file oriented and expects a server to
      do things in terms of open files.

      UNIX treats open files as something that lives 
      above the VFS interface and file system implementations
      don't have any knowledge of the open-file associated
      with a given operation.

This combination makes it difficult to cleanly resolve the
issue.  If we keep the protocol essentially open-file-instance-
unaware, then NT servers have no way to properly associate 
individual locks with open-file-instance's.  This can cause 
problems in some cases when it is necessary to close an
open-file on the server side, if the lock has been associated
with an open-file which is being closed.

On the other hand, if we were to change the protocol to make
open-file-instance uniformly visible (by making stateid per-
open-file-instance), UNIX clients would have a problem.
The typical VFS interface does not pass open-file information
through to the implementation.  Without major changes to the
VFS interface (real hard to do for many people), they couldn't
conform to the protocol.

I don't think there is an elegant solution to this problem.
I think we can have a reasonable compromise, that addresses
the NT server issues but provides the ability for UNIX clients
to function without additional knowledge of open-file-instances.
This does leave open a few cases in which an open-file-unaware
UNIX client may get some errors on a downgrade from an NT
server, when it hasn't provided complete information about 
locks.

Here is what I think should be done:

     Modify open-downgrade so that the new access level must
     be exactly equal to the union of some subset of the
     OPEN's previously done.  This is an SMOW (simple matter
     of writing).  Servers may return an EINVAL error where
     this condition is not met, but don't have to, so servers
     which don't deal in open-file-instance's don't have any
     additional burden.

     Leave stateid's as they are since UNIX clients are not well-
     equipped to deal with a open-file-centric protocol.

     Provide a way that a client can indicate which open-file
     a given LOCK request is associated with.  (My current favorite
     is the seqid of the associated open request). This information,
     when present, can used to properly assign byte range locks to
     open files, avoiding any problem on downgrade.

     Provide a way for clients which cannot provide the open-file
     information on a LOCK request (i.e. UNIX clients with unreformed 
     VFS interfaces), to indicate "don't know" (e.g. special flag or
     seqid value of all ones).  In this case, a server is allowed
     to guess in assigning the lock to a particular open-file-instance.

     On a downgrade which reduces the access level such as to eliminate
     that established by some set of previous OPEN's, the server MAY
     (NT servers will) return an error, NFS4ERR_LOCKSLEFT if any
     locks are still associated with the open-file-instance being
     closed (implicitly) as a result of the downgrade.  Servers
     don't have to return this error, so they aren't required to 
     keep track of separate open-file-instance's if they have no
     other need to do so.

     The above paragraph applies as well to cases in which the client
     doesn't tell the server what open-file-instance a given LOCK
     belongs to.  In this case, the server may choose arbitrarily,
     with the result that a downgrade may fail (with NFS4ERR_LOCKSLEFT)
     when, if the server had the correct information, it would not
     have failed, but there is nothing the server can do deal with 
     that.

     In the situation noted above, it is the responsibility of the
     client to deal with the NFS4ERR_LOCKSLEFT error, typically by
     punting on the downgrade and leaving the current access greater
     than it needs to be until a CLOSE is actually done.

The deferral of the downgrade can result in some opens from other 
lockowner's being denied when they should not be, but I think the
problem is small, since:

     The error is in the right (safe) direction.

     In a totally DENY=NONE (i.e. UNIX) client environment, no
     problem is perceivable by other clients.

     Where the client encountering the problem is DENY=NONE, the
     server has a way of assigning locks in the don't-know case
     which very unlikely to pose a problem for access by other
     (e.g. NT) clients.  See below.

     There are no problems except in cases in which there are
     multiple opens with different access modes of the same file
     by the same lockowner *and* byte-range locking is used at 
     the same time.

     UNIX clients have a way of avoiding the problem if it is 
     an issue for them (when they operate over nfs-v4 in a mixed
     UNIX-NT environment), if they enhace their VS interfaces to
     pass the necessary information.

Here is a suggested algorithm for assignment of don't-know locks
in a DENY=NONE environment.  Generalization to handle all deny
modes is left as an exercise.  There are four combinations of
access modes to deal with: READ/READ-WRITE, WRITE/READ-WRITE,
READ/WRITE, and READ/WRITE/READ-WRITE.

     In the first two cases, the server assigns don't-know's
     to the open-file-instance whose associated mode is a 
     subset of the other.  Since there is no downgrade needed
     to transform READ/READ-WRITE to READ-WRITE alone (since
     they both evaluate to READ-WRITE,DENY=NONE), and a downgrade
     to READ alone or WRITE alone will not run into the
     problem, everything is fine.

     In the third case, the server assigns don't-know's to 
     the open-file-instance hose mode is READ.  On a downgrade
     to READ (WRITE instance gone), no problem will be encountered.
     A downgrade to WRITE (READ instance gone) may spuriously
     get a NFS4ERR_LOCKSLEFT but the effect of ignoring this only
     affects another client doing an open specifying DENY=READ
     (i.e. someone who doesn't care if someone is writing the
      file, but can't open it if someone else is reading it).
     Denying an otherwise grantable open in this case, seems 
     fairly minor, and, its the best that can be done if UNIX
     clients don't give the server the necessary information.

     The last case is a combination of the two above,  don't-know's
     are assigned to the READ instance.  Based on the sequence of
     close on the client, you either fall into the logic of 
     one of the first two case, in which there is no problem,
     or the third case, in which there is a very minor problem.

---

From: RJ Atkinson <[email-masked]>
Date: Sat, 18 Nov 2000 08:02:50 -0500

From a user/operator perspective, it would be better 
to have limited capabilities on a new OS platform (that is
broken wrt NFS) than to change things such that other OS 
platforms (that well work today and for many many years) 
might have increased chance of operational problems.

	I'm not sure how to translate the above into the gory
details, so I won't try and leave that as an exercise to 
the smarter folks here.

	Generally speaking, NFSv4 looks gratuitously more
complex than NFSv3 to me.  From the sidelines, it appears
this was driven by a desire to support an OS that doesn't
really support the right kinds of semantics for NFS.  Much
though I prefer NFS to SMB, it really might have been better
to leave that other OS with a somewhat limited NFS capability 
and keep the protocol simpler for the existing installed base.
I really fear having to troubleshoot byzantine corner cases
operationally that exist in NFSv4 but don't exist in NFSv3
(with existing OS platforms).

Regards,

Ran
[email-masked]

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 20 Nov 2000 07:39:51 -0800

> might have increased chance of operational problems.

It certainly would make life simpler, but it's been a 
goal all along not to be as UNIX-centric as NFS has been
in the past.

> details, so I won't try and leave that as an exercise to 

Without getting into gory details, the one thing that is
particularly worrisome in this regard is volatile file
handles.  I think everything else is in the noise.

We need a whole lot of testing with servers providing volatile
file handles under real-life conditions.  Even then, I fear
the distribution of problems will have a very long tail 
(as handles expire at just the wrong times) that users are 
going to wind up having to confront.

> complex than NFSv3 to me. 

As far as what is truly gratuitous, as opposed to being
reasonably related to some real goal that we may (or may 
not) later think was ill-advised, I only know of two things, 
and neither adds all that much complexity:

     TAG's

     Multi-component pathnames in LOOKUP and OPEN

> From the sidelines, it appears
> really support the right kinds of semantics for NFS. 

There is certainly some of that but I don't see that as the
major source of v4's complexity.  The biggest reason for v4's
increased complexity is that v4 has to be stateful.  This follows
from the decision to unify all the NFS-associated protocols
including NLM into NFS-v4.  Once you do that, you're "not 
in Kansas anymore". 

There is certainly complexity that comes from the attempt to
be less Unix-centric:

     The attribute model

     Share reservation semantics in OPEN

     Volatile file handles
  
> Much

It might have been.  If we had a smaller set of goals, we might
have been done by now.  The trouble is everybody has a different
candidate on what to throw out.  The good news is that we have
converged on a set of goals, we have a protocol which makes
sense and implementation is actually going on.

> (with existing OS platforms).

To avoid that, we have to do better in interoperability testing
up front.  We need to have a far richer set of interoperability
tests than we currently have.  Also, we have to take acount of
all the features of v4 that give clients and servers the opportunity
to Express Their Individuality in ways that may cause problems
when they try to interoperate:

     Clients can produce arbitrarily complicated COMPOUND's

     Servers can can stop processing ops within a COMPOUND when
     they choose.

     Servers can choose to not support attributes that everyone
     else supports.

     Servers can expire file handles whenever they choose.

     The fact that Unix and NT clients and servers will behave
     in accordance with their own underlying natures (they will
     do everything entirely differently whenever the protocol
     allows them to do so)

I think we need tests that systematically expose the complete
range of possible client behavior to a server and vice versa.
We have a more complicated protocol with a greater range of
implementation variation possible for the clients and servers.
Unless we seriously address the interoperability issues, users 
are going to have to deal with them and they are going to object, 
justifiably so.