Subject: minor versioning revisited

From: Mike Eisler <[email-masked]>
Date: Thu, 4 Nov 1999 13:46:02 -0800 (PST)

As Spencer has noted previously, we need closure on minor versioning.
I humbly offer my head on the platter again, with a new proposal, and
hopefully crisper than the last one I offered years ago. For what it is
worth, one of the Sun folks who hated the minor versioning idea 3 years
ago, said he liked this one.

rules for minor versioning
--------------------------

A. To make life easier for customers, the protocol should encode a minor
version # that corresponds to a published RFC.

	There have been objections to adding minor version number
	encoding overhead to the protocol. In my experience, customers
	simple hate it when they get a product that is versionable, but
	there are no tools to determine the version. It makes support a
	nightmare if you can't easily establish over a telephone
	conversation what version the customer has. If anything, this
	issue is more important to nail down with network products than
	stand alone products.

B. Minor versions do not add/delete procedures

	There was early objection on the alias around
	a minor versioning proposal that did so. The objections
	can be summarized as:

		1. RPC already has a mechanism for adding/deleting
		procedures; change the version field in the call header
		and then change the set of procedures.

		2. PROCUNVAIL error handling is more complex when
		NFS V4 can add or delete procedures.

		3. Perhaps NFSv4 should be a suite of multiple
		program numbers, each of which can be versioned
		using the normal RPC convention via the program's
		version field in the call header.

	One and two taken together simply mean that we shouldn't
	add RPC procedures in an existing program/version pair,
	since it is rather unorthodox.

	Point number three was never supported by a concrete
	proposal. Since then the NFSv4 protocol has collapsed to
	just two procedures, NULL and COMPOUND. Since everything
	is done with COMPOUND, adding/deleting procedures is not
	an issue.

C. Minor versions can add operations to COMPOUND.

	Since everything is done under COMPOUND, there is no impact on
	an implementation's RPC framework. There's nothing unorthodox
	about doing so ... it's merely a switched union. A precedent
	is how well the RPC framework copes with adding security
	flavors to via the RPC header's AUTH credential and verifier
	fields.

C.1 Minor versions can append attributes to GETATTR4args, bitmap4,
and GETATTR4res.

	Naturally. However ...

C.2 Minor version X must append any new attributes after the last
documented attribute.

	Suppose we add a mandatory attribute and stick it in the
	first bitmap word. Given the way attribute results are
	specified, as an opaque array of per-attribute XDR encoded
	results, packet sniffers for minor version X-1 will have an
	impossible time parsing the bit stream. We should not
	negatively impact the ability to diagnose problems.

D. Minor versions cannot modify the structure of an existing
	operation.

	It's going to get messy to have to deal with N different
	variations of something like OPEN, which is already quite
	complex. Note that this rule doesn't mean we can't add bits to
	flag fields, such as new attributes to GETATTR's bitmap4 data
	type.  The point is to preserve backward binary compatibility on
	the wire.

	The side benefit is that a packet sniffer written for minor
	version X can do a reasonable job of parsing operations for
	minor version X+1, albeit with a warning to the user that
	not everything can be parsed.

D.1. Minor versions cannot modify the structure of an existing
	attribute.

E. Minor versions cannot delete operations.

	Think it through and you find this is a corollary to
	rule D above. If you can delete an operation in minor version X, then
	what's to prevent version X+1 from introducing a new operation
	in the same slot with same opcode value? Of course, you
	could have a rule that states that if an opcode is reused,
	it has to use a compatible struct, but that is error prone.

	What about deleting flag bits? In principle, except for
	oddities like bitmap4, this might be all right. However before
	the WG decides to do so in a particular minor version, it
	should consider the implications.

E.1 Minor versions cannot delete attributes.

F. Minor versions can declare an operation as mandatory to not implement.

	Suppose in minor version X operation Y is declared obsolete. This
	means a client should not send it, and a server should return
	an NFS error as a opposed to dropping the request as an XDR
	decode error.

	This is a way to delete the *support* of an operation, but
	still preserve its structure definition so that future minor
	versions can unobsolete the procedure if necessary.

	There's precedence for this. Recall the WRITECACHE procedure of
	NFS Version 2. It's there in the specification, and implementors 
	are told that they shouldn't implement it.
	
F.1 Minor versions can declare attributes mandatory to not implement.

G. A client and server that supports minor version X must support
minor versions 0 through X-1 as well.

	Otherwise, it's an interoperability nightmare. The counter argument,
	what if one particular minor version is so bad that one would
	rather not support it? First, since the minor versions have to
	got through the IETF process, one would hope that due diligence
	would avoid this. Second, if version X is so bad, if if version X-1
	was considered pretty good, then it's not rocket science to publish
	a version X+1 that is a clone of X-1. Business considerations will
	encourage implementation X+1.

H. A client MUST NOT attempt to use a stateid, file handle, or similar
returned object from a COMPOUND with version X for another COMPOUND
with version Y, where X != Y.

proposal
--------

I. Add a minor version field to COMPOUND4args. Initially supported value
is 0.

II. Add an NFS4ERR_MINOR_VERS_MISMATCH error status.

	When the server gets an unsupported minor version, it MUST
	return a COMPOUND4res with a zero length operations result
	array

III. Reserve operation number 2 as an operation we can use for version
negotiation on the day when we actually have more than 1 minor
version.

	Why not solve that issue today?
		Because we don't have to and have some real work to do
		to get minor version 0 out the door.

		Because if we attempt to we will probably get it
		wrong.  Since we can't predict how frequently we'll
		make minor revisions, nor how much delta each change
		will be, it seems likely we'll either over engineer or
		under engineer.

	What might the operation look like?

		In the future, operation 2 might be as simple as one
		with void arguments in the operation, and the highest
		supported version in the COMPOUND4args. The server
		returns an NFS4_OK if supported,
		NFS4ERR_MINOR_VERS_MISMATCH, if not.

		Or it might be as complex as the client sending an
		array structs, each element representing a minor
		version. Each element would contain a bit mask of the
		client's supported operations, which the server would
		logically AND with its supported operations.

		The operation itself might have its own version number
		independent of the NFS V4 minor version field to allow
		us to develop more flexible schemes.

		More than likely it will be something between the
		extremes.

	How does a NFSv4 client that supports minor versions > 0
	detect that the server supports only version 0?

		We do have to specify this aspect of version
		negotiation protocol now.

		When the client issues a COMPOUND request with say
		minor version 5 in the COMPOUND4args, the server
		returns a zero-length operations result with the error
		NFS4ERR_MINOR_VERS_MISMATCH. The client has two
		choices:

			1. re-issue the COMPOUND with a lower minor
			version.

			2. issue a COMPOUND with opcode 2, and the
			minor version field set to 1 or higher.

		In either case the server will return
		NFS4ERR_MINOR_VERS_MISMATCH.  If the minor version
		number in the compound arguments that generated the
		NFS4ERR_MINOR_VERS_MISMATCH was 1, then the client
		knows definitively that the server supports just minor
		version 0.

	Once we add minor versions, won't clients have to "guess",
	possibly several times?

		"Today" all clients support minor version 0. Thus those
		clients will never need to guess, because they never
		get a mismatch error, because V4 servers will always
		need to support minor version 0.

		"Tomorrow" there might be a client that supports
		version 10. It will first issue a COMPOUND with minor
		version set to 1, and opcode set to 2. The result will
		either be a version mismatch, meaning that the server
		supports just minor version 0, or a success, with
		sufficient information to indicate what version the
		server does support.

---

From: Brent <[email-masked]>
Date: Thu, 04 Nov 1999 23:59:35 -0800

I like Mike Eisler's rules for minor versioning.

However, I'd prefer not to see the minor version
required in the COMPOUND4args.  I think it's simpler
as an attribute:

	minor_version	The minor version number of the protocol.
			The first version is number zero. 

The client can query the server's minor version anytime
with a GETATTR.  Most likely on the first call (at mount time)
along with lots of other server information.

The client could use a VERIFY operator to abort a compound op
if the server's minor version is too small.  For instance,
if the server was to use a compound op that required version 2
then it could prepend a "VERIFY minor_version 2" (I'm assuming
the server would evaluate this attribute comparison as
an inequality rather than as a strict match on equal).

There's an advantage for client's that use only minor
version 0.  They don't need to query minor_version or
VERIFY it - nor do they need to put the version number
in the COMPOUND4args.

	Brent

---

From: Mike Eisler <[email-masked]>
Date: Fri, 5 Nov 1999 06:11:23 -0800 (PST)

> required in the COMPOUND4args. 

Then how does the server know what semantics the client wants
for an operation? Minor version X+1 could be jsut like X, except that
it might declare a certain attribute that was recommended as mandatory. Or
it might slightly change the meaning of a flag bit.

> an inequality rather than as a strict match on equal).

I'm envisioning servers that will support different minor versions on
different files and I'm getting queasy.

	-mre

---

From: Neil Brown <[email-masked]>
Date: Mon, 8 Nov 1999 11:55:11 +1100 (EST)

Sorry of the 'dribs and drabs' approach, but one more, fairly
significant, change that a minor version might introduce has occured
to me.  It is error return values.

A minor version should be allow to both
  - refine error statuses (where before NFS4ERR_X, you now might get
    NFS4ERR_Y which tells you something more precise) and
  - introduce new error states (This would have worked before, but now
    I am telling that there is an error so you can recover in the new
    way that we agreed on).

An interesting way to explore minor versions is to suppose that we
hadn't included some particular feature in version 4.0.  What would be
needed to add it to 4.1.  To add filesystem migration, you need to add
the error state NFS4ERR_MOVED

Hopefully that is the last.

NeilBrown

---

From: Mike Eisler <[email-masked]>
Date: Mon, 8 Nov 1999 10:27:38 -0800 (PST)

>     I am telling that there is an error so you can recover in the new

I quite agre, and had the same thoughts Sunday morning as I
was preparing my ietf presentation.