Subject: Feedback on draft-ietf-nfsv4-01 - stateid

From: Neil Brown <[email-masked]>
Date: Mon, 11 Oct 1999 14:16:10 +1000 (EST)

My forth and (for the moment) last email with feedback on
draft-ietf-nfsv4-01

NeilBrown

Stateids In General:

The whole stateid concept seems a little complex to me (it took me
several attempts to wrap my mind around it).

Two fact that only became clear after repeated readings are:

 1/ There are two different sorts of stateids.  One for
    OPEN/LOCK/CLOSE, which is per-lockowner, and one for delegations
    which is per-client.
 2/ stateids progress from state to state. Each LOCK/UNLOCK/CLOSE call
    takes an old stateid which becomes invalid, and returns a new
    stateid. (in a way similar to the seqid of OPEN/LOCK/LOCKU. Why
    doesn't CLOSE need a seqid?)

Given these two facts, it all falls into place and makes sense, so
stating them clearly early in the document might help.

However, in trying to understand the usage of stateids is spent quite
a while trying to figure out what was really needed and what the
important issues are, and my conclusion is that the protocol is
current more complex than is necessary.

The complexity comes from that fact that there are three sort of
values that relate to state, and for a given client there are multiple
instances of these values.

The values are:

    LOCK stateids which are 
	 per-state, per-lockowner, per-file
    DELEGATE stateids which are
         per-file, per-client and per-state
    seqids which are
         per-lockowner, per-state

My thought is: why are these three needed, and why such fine
granularity?  Why not use just one stateid per client which changes
whenever there is a state change.

Some of the issues:

  Per-lockowner information is not needed.
    I have had great trouble understanding why there needs to be a
    concept of a lockowner.  The way I see it, the NFS client serves
    as an agent for a number of applications which each do IO and
    locking etc on various files.  However this fact is not any
    business of the server's.  If two applications try to lock the
    same region of the same file, the NFS client can detect this,
    block one of them, and make a lock request on behalf of the
    other.  When that application releases a lock, the client will
    release the lock with the server and, if the second client is
    still waiting, attempt to reclaim the lock, now on behalf of a
    different application, but why should the server care.

    In short, the substructure of the client should be opaque to the
    server. 

    I can see two objections to this, neither of which hold much water,
    but I will mention them anyway.
     - section 8.6, para 2 suggests that a server admin might revoke
        locks held by a particular lockowner. For this to happen, the
	protocol needs to know about lock owners.  This seems very ad
	hoc. Revoking locks for a given file or a given client sort of
	makes sense, but for a given lockowner? No.

     - deadlock detection needs to know about entities that will not
       release a lock while they are trying to claim another lock.
       Thus deadlock detection would need to know about lockowners.
       But NFSv4 doesn't seem to support deadlock detection.

  Per-file state information is not needed.
    The only reason that I can think of for keeping per-file stateids
    is to ease multi-threading in the client.  read/write requests must
    use the "current" stateid and so it seems that they cannot
    reliably be run in concurrently with any open/clock/lock request
    which might change the stateid.  If there is a different stateid
    per file, then different files can be processed in parallel quite
    separately. 
    I think this could be answered with a short stateid history on the
    server.  If the server remembered the last half dozen stateids and
    the files affected by them, then any IO request on any other file
    could be allowed through with an old stateid.  A history
    length of 6 would allow there to be 6 concurrent  state changing
    requests without interfering with parallel IO on other files.
    If a request arrives with a stateid that has fallen off the
    history, an appropriate error would be returned and the client
    would retry with a more-uptodate stateid.

  Does the OPEN stateid have to be distinct from the DELEGATE stateid.
    No, I don't think it does.  Providing the client and the server
    both agree as to what sequence of operations has lead to a given
    state, then any operations, OPEN, CLOSE, LOCK , DELEGATE,
    DELEGRETURN, or whatever be be implied by a given state.

  Can stateid's be used to provide the strict sequencing that seqid
    gives?

    Yes.  Per-file stateids cannot because an application on the
    client may require sequence of locks on different files.  However,
    with a per-client stateid, the client is able to say "this request
    must come after that request".  Also, the "consecutive integers"
    requirement of seqid is much stronger than is needed.  All the
    server needs to know is what was the "last" number and what will the
    "next" number be.  With stateids, the "next" number is whatever
    stateid was last returned, and the "last" the previous one.

To summarise my proposal:
  Upon successful SETCLIENTID, a stateid is given to the client by
  the server.  This is unique across all clients and across all
  (reasonable) time.

  Each state changing operation (OPEN, CLOSE, LOCK, LOCKU, DELEG*)
  takes a stateid and, if successful, returns a new stateid.  The
  stateid that is passed must be the "current" stateid or, at the
  servers option, a recent stateid since which the relevant file has
  not suffered a state changing operation or any IO from the client.
  With the new stateid, the server returns the number of state changes
  since the passed stateid, so that the client can be sure, in the
  face of multiple concurrent state changes, which is the most recent
  stateid. 

  Each operation which may be affected by state (read/write) must
  include a stateid. This stateid must be the "current" stateid or, at
  the server's option, a recent stateid since which the relevant file
  has not suffered a state change from the client.

  This server might record recent state by:
   -  recording the most recent few stateid's for each client, and the
      file which was affected by the last change for each
   -  recording with each file the stateid and client at the last IO
      operation. 

  Given this, all locking state for a given client-server pair is
  represented by one stateid.  This seems reasonable as all of that
  state is treated as a whole for both crash recovery and lease
  extension. 

  It is possible that the number of old stateids that the server will
  accept could be negotiated in the SETCLIENTID request, though it is
  also possible that the server could dynamically determine the best
  number, and the client could dynamically determine when it has hit a
  hard limit.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 13 Oct 1999 12:00:49 -0700 (PDT)

>     different application, but why should the server care.

The client that wants to do all this work may: simply set the owner field
to a zero length string. It isn't much extra work for the server to arbitrate
access on an nfs_lockowner that contains both a clientid and an owner field
versus one that contains just a clientid field. Since the server has to
deal with multiple clientid's coming from the same client host (consider
the scenario of a user-level client co-existing with a kernel-level client
in an NFS file system), nfs_lockowner needs to include the clientd anyway.

Arbitrating access based on the client's source IP address is going to be
messy because the client might have multjple IP addresses or it might be using
dynamic IP address assignmnent. The nfs_lockowner and clientid abstractions
simplify the server implementaton.

For simple user level clients, that might consist of multiple processes using
the same suer id and clientid, letting the server do the work simplifies
client implementation.

Without thinking it through, I suspect that continuing the model of multiple
nfs_lockowner's per client means that your goal of a single
stateid per client cannot be achieved, but perhaps I'm wrong. If you
haven't already entered into a dialog with David Robinson (designer of
the locking/share protocol), and Dave Noveck (designer of the
delegation protocol), I suggest you do so. 

(BTW, the server containing the nfsv4-wg alias is unreachable right now 
due to a routing problem). This means that it will be some
time before the responses I've written will get to the entire
alias).

	-mre

---

From: Neil Brown <[email-masked]>
Date: Thu, 14 Oct 1999 13:06:55 +1000 (EST)

> delegation protocol), I suggest you do so. 

Yes, the client could present a single nfs_lockowner if it wanted to,
but should the protocol require the server to deal with multiple
nfs_lockowners per client if doing so complicates the protocol?

Yes, we can't trust client source IP, the "Client" concept is still
very important.

Your "simple user level client" scenario violates the "smart client,
dumb server" principle, which admittedly is not cast in concrete.  A
"simple client" could present different "client ids" for the different
processes if it wanted to.  It would increase it's lease-extension
workload, but that's it's choice.

Two other issues have occurred to me since I first presented the
single-stateid-per-client idea that are relevant:

 1/ How does the server let the client know that it has revoked
    the state for a particular file (e.g. revoked a delegation or just
    possibly revoked a lock).
    With stateid-per-file, it was clear: 
	 any use of the relevant stateid is replied to with
         NFS4ERR_EXPIRED 
    With single-stateid, this must become:
         any use of the relevant stateid together with the relevant
	 filehandle results in NFS4ERR_EXPIRED
    However that means that the server must remember all the
    filehandles for which is has revoked state.
    Alternatively, the client could assert in each request "I believe
    I have state for this filehandle" if that is the case.  The server
    could then reply EXPIRED if doesn't current have any state for the
    filehandle with that client.
    If the protocol mandated opening before access, which the draft
    seems to suggest but is not entirely clear on, then making any
    "state related" request (read/write/lock etc.  not getattr) would
    implicitly assert "I believe I have state", and would be replied
    to with NFS4ERR_EXPIRED (or similar) if the file hadn't been
    OPENed.

    This assumes that the server can only revoke state on a
    file+client basis. i.e. it cannot revoke a delegation but still
    leave an open active.  I don't see this to be a big cost as
    revocation only happens when the client is not behaving its self.

    (I feel that discussion isn't very clear.  Maybe I should write a
    "State" chapter to put my ideas in a coherent format.)    

 2/ Who "owns" the stateid, in terms of what credentials do you need
    to change it (i.e. request a state change from this state)?
    Possibly more significantly, who "owns" a seqid in the current
    proposal? 

    The seqid or stateid needs to be protected otherwise a denial of
    service is possible.
    e.g., given the current proposal, a rogue client determines in
       some way a valid stateid for a given handle and a given
       nfs_lockownr (possibly by guessing, possibly by snooping), and
       then makes lots of lock requests with stateid/filehandle and
       differing seqid's.  Then all the bad seqids will be quietly
       ignored (the spec says so) and when the right one is hit, the
       server and the client go out-of-sync and cannot continue.

    The obvious remedy here (which couldn't apply if we dropped
    nfs_lockowners) is to require each lockowner to present with the
    same credential every time, so a particular principle could only
    hurt themselves.  This might work, but it feels restrictive.

    The only alternative that I can think of requires stateids to be
    "owned" by the principal in the credential that came with the
    SETCLIENTID.  This would mean for open requests (for example), you
    would need 2 principals to be authenticated in the credential: the
    principal who owns the client, and the princple who wants access
    to the file.  But can a credential identify two principals? I haven't
    read enough aboue RPCSEC_GSS and GSS yet to be sure, but I doubt it.

    (ASIDE:
     actually, this is a bit like chgrp in Unix, which requires you to
     own the file and be a member of the group, so you need two
     credentials.   AUTH_SYS allows you to send multiple credentials,
     pretending that it is one big credential.  How does KerberosV
     cope with the possibility of a "newgrp" command? Does it?)

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Thu, 14 Oct 1999 04:07:40 -0700 (PDT)

> nfs_lockowners per client if doing so complicates the protocol?

I don't think it complicates the protocol. The server has to deal with
an arbitrary number of client ids. All the nfs_lockowner struct does is
wrap the client id, plus some variable length information. It makes
byte compares a little longer. 

> dumb server" principle, which admittedly is not cast in concrete.  A

If there are two different client hosts, the server has to be smart
enough to arbitrate access between the two. SInce the server has
to implement this logic, implementing the server to arbitrate access
among multiple processes on the same client is trivial.

> 	 any use of the relevant stateid is replied to with

Section 8.4 states:

	"any stateid for a
   client will renew all leases for locks held with the same client
   field."

As section 8.5 and its subsections describe, if the client loses a lock,
or all its locks, then any I/O will get a NFS4ERR_GRACE or NFS4ERR_EXPIRED.

>        server and the client go out-of-sync and cannot continue.

In order to do I/O or locking, the client has to have the appropriate read or
write access to the file as determined by the client's credentials, the
permission bits, file owner, group owner, and ACL. This is only an issue if
the client is using junk like AUTH_SYS and the server allows AUTH_SYS.  

As an attacker if I have read access to the file, then I don't need to mess
with the stateids and seqids to deny you write access to the file. All I need
to do is get a read lock on the whole file, and go on holidays. And if you've
given me write access, I won't even bother with holding write locks for a long
duration to deny you service ... I'll just write trash into the file.

>      cope with the possibility of a "newgrp" command? Does it?)

The Microsoft version of Kerberos V5 allows this by including user id and
group id information in the ticket from the KDC. (However the method by which
this information is encoded is, to my knowledge, non-standard, and not
documented). The KDC thus becomes both an authenticator and authorizer.
Generic versions of the Kerberos V5 KDC tend to stay away from operating
system specific concepts like uids and gids. So, if using say Solaris7, with
the Solaris Kerberos V5 (SEAM 1.0), newgrp won't work for you, because the NFS
server just gets your principal name, and maps you to uid and gid, and gid
list that the NFS server determines from the passwd and group databases.

When you raised your question, I was surprised to find that Solaris supports
newgrp. I'd assumed for the last 16 years that use newgrp had been
replaced by the much more sensible multiple group stuff that
4.2 BSD introduced. Does Linux support newgrp?

NFS over AUTH_DES works the same way as NFS over KerberosV5. NFS/AUTH_DES
is over 10 years old, and I've never come across a customer complaint
about the newgrp issue. 

RPCSEC_GSS is flexible enough to support newgrp semantics. It's just
a small matter of finding a GSS-API mechanism to plug into it. I think
we should stick with generic stuff, to increase the opportunities for
interoperability in heterogeneous environments.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Fri, 15 Oct 1999 13:45:55 +1000 (EST)

> interoperability in heterogeneous environments.

Linux does support newgrp, though it probably isn't used much.  I was
just trying to think of a current API that didn't match the services
provided.

I guess that if the security service used doesn't support something
like newgrp (or user-selected capabilities in general), then there is
nothing NFSv4 can do about it, and if the security service does
support it, then NFSv4 would just treat a different security context
as a different nfs_lockowner.

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Fri, 15 Oct 1999 09:06:51 -0700 (PDT)

> as a different nfs_lockowner.

I don't see why.

The nfs_lockowner is simply a unique identifier of the thread doing the
locking. If a thread issues two different lock requests, with two different
principals, then the same nfs_lockowner will likely be used. The issues the
server are interested in is:

	- is the clientid field in nfs_lockowner valid (the server
		generates the clientid in response to a SETCLIENTID
		request)? There's no requirement that the principal that
		issued the SETCLIENTID be the same as the principal that
		issues a lock request using the lock request. 

	- does principal have authorization to perform the lock operation?
		For a UNIX server, this would mean, does the principal
		map to uid or gid that has the appropriate read or write
		access to the file?

Keep in mind that in the UNIX world, the process id plays a big role in
arbitrating lock accesses, and in UNIX world, a process can switch uids at
will. This is part of the motivation behind the "owner" field in the 
nfs_lockowner structure: with pre-NFS V4 locking there was no way to tell the
server the process id of the process, and so mandatory locking was not
implementable. 
 
		-mre