Subject: Re: [nfsv4] Fwd: I-D Action: draft-ietf-nfsv4-versioning-05.txt

From: [email-masked] (J. Bruce Fields)
Date: Wed, 17 Aug 2016 14:58:46 -0400

On Thu, Jul 28, 2016 at 05:54:03AM -0400, David Noveck wrote:
> I believe this update addresses all pending comments regarding -04.  Thanks
> to Mike Kupfer for his review comments.
> 
> In the next few weeks, I will be posting a -06, which adopts a somewhat
> different approach to the versioning problem.
> 
> Based on the discussion at IETF96, I have decided that it is
> important to have a shorter document available soon, to support our needs
> to proceed with such pending extensions as xattrs and umask.
> 
> Draft-dnoveck-nfsv4-extension-00 provides an example of a
> document taking such a more constrained approach and I expect that -06 will
> be similar to that in its overall structure.
> 
> In producing -06, I will try to address any comments I receive on
> draft-dnoveck-nfsv4-extension-00.  In order to make sure the group is in
> agreement regarding our transition to a different extension model, I'm
> particularly interested in people noting:
> 
>    1. Anything in draft-dnoveck-nfsv4-extension-00 that you feel is
>    informational in nature and thus inappropriate in a standards-track
>    document.
>    2. Anything else that you feel should be pruned to make the document
>    shorter and easier to get approved.
>    3. Anything in draft-ietf-nfsv4-versioning-05 that is not in
>    draft-dnoveck-nfsv4-extension-00 that you feel should be retained in -06.
> 
> My underlying assumption is that the group is agreed that we need to adopt
> a more incremental extension model.  If there is anyone who disagrees, it
> would be better to raise this issue now and discuss it, rather than waiting
> for WGLC on draft-ietf-nfsv4-versioning.

Comments on draft-dnoveck-nfsv4-extension-00:

I appreciate that it's a lot shorter, thanks, it's much easier for me to
deal with than draft-ietf-nfsv4-versioning-05.

The goal is to preserve interoperability on protocol upgrade: two
implementations can choose to implement any set of extensions they'd
like, and from the user's point of view everything should behave no
worse than if both implementations supported only the parts of the
protocol they share.  I think everything follows from that, and the
extent to which it's spelled out here still seems like mild overkill.

I wonder if this would work better as an NFSv4.3 document that specified
an initially empty but extensible minor version:

	- we could specify once and for all the correct errors in the
	  face of unknown protocol, instead of having to take into
	  account every possible legacy behavior in situations that were
	  previously unspecified.
	- it wouldn't add much additional burden for the umask or xattr
	  proposals--updating our prototypes to use 4.3 will be easy.
	- it might help focus the document as making it primarily about
	  how this one minorversion is going to work rather than being
	  about versioning in general.

Anyway:

I think section 2.3 could be summarized as "implementations may support
some versions and not others".  I'm not sure it's necessary to write all
this out.

I understand the unknown vs. unsupported distinction, but I don't really
understand what benefit we get from it.  How would a client, for
example, behave differently if it discovered a server didn't know of a
certain extension, vs. whether that server didn't support it?

End of p. 11, from "requesters can ascertain the knowledge...": this is
an awfully verbose way of saying that implementations determine support
by looking at rpc requests and replies.  Let's just cut those four
bullet points.

5, 5.1, 5.2: I don't follow this section.  It appears to be only about
protocol extensions that are possible only with new minor versions?
Does that belong in a document about extending minor versions?  If so
maybe it could be merged with section 7?

5.2: COMPOUUNDS->COMPOUNDS

The term "minor version group" is used a few times, but not defined.  I
think it's out of scope here and we should just remove the uses.

8.1 is useful to state, I'm a little surprised the requirement on minor
versions staying constant over lifetime of a clientid hasn't already
been stated elsewhere.  Is there an erratum?

8.2: "to provides"->"to provide", "by a searching"->"by searching".

8.2, final four bullet points: sounds like perfectly reasonable
behavior, but I don't really understand the motivation--what would
happen if we got these errors wrong?  These are cases only a buggy
client would hit, if I'm understanding right.

9: "the working will"->"the working group will"

"the indexing material": I think you're referring to some stuff that got
removed.

"In other cases, it is best if client or server behavior needs to change
in a way which raises interoperability concerns."  I don't understand
this sentence.  I'm not completely sure about the following sentence
either.

I'm not sure section 9 is entirely necessary.  I think this was
motivated by the possibility of backporting the umask extension to 4.1
or 4.0?  I'd rather deal with these concerns when we do that.

--b.

---

From: David Noveck <[email-masked]>
Date: Thu, 18 Aug 2016 12:52:41 -0400

> deal with than draft-ietf-nfsv4-versioning-05.

I'm OK with basing -06 on draft-dnoveck-nfsv4-extension-00, with
some changes based on your suggestions.

Before I do that, I need to make sure the working group as whole is
OK with this shift.  Let's suppose I'll submit a -06 soon after Labor Day.

> protocol they share.

That;s a good way to put it.  I think I'll put something like that in the
introduction.

> I think everything follows from that, and the

Perhaps it is to members of the group.  But what i'm worried
about a situation in which I'm dealing with a possibly
uncomprehending IESG.  Given that, I'd prefer mild overkill to
any sort of underkill.

I lot of people have trouble understanding how implementations
of two XDR files that not  the same can successfully interoperate.

> an initially empty but extensible minor version:

No, but I think we could define such a version as an additional think to do.

>          account every possible legacy behavior in situations that were

If we did that, we would invalidate existing behavior, which makes things
cleaner,
but also limit interoperability.  For now, I would prefer to make sure we
could
be maximally interoperable.

>          proposals--updating our prototypes to use 4.3 will be easy.

Not updating them at all would easier still -)

As far as the specs, I think xattrs doesn't mention the target version while
umask mentions v4.2.  I prefer xattr's appraoch since it is more flexible.

 >        about versioning in general.

But right now we have a document which is (in part) about versioning in
general
and it is RFC5661.   We can either contradict that in general or create a
new
exception for v4.3.   I'd rather not do the latter.

> some versions and not others".  I'm not sure it's necessary to write all

Actually it also give definitions of terms used throughout the document.

> certain extension, vs. whether that server didn't support it?

What is different is what it is entitled to conclude about the the other
elements
of the protocol.  If a protocol element is known or unknown, the client is
entitled to assume that all other protocol elements in the same feature
package have are the same.  On the other hand, if you find out that it is
unsupported, you can't assume that.  For your particular extension, you may
be
able to identify these two concepts but the intention is to be able to do
support
more  complicated
extensions as well.

> by looking at rpc requests and replies.  Let's just cut those four

I'm pretty sure that if  I did, the IESG would ask, "exactly how can they
do
that?'" sigh!

> protocol extensions that are possible only with new minor versions?

It is.

> maybe it could be merged with section 7?

But this document isn't just about that.  It is about how the NFSv4 can
be extended.  It changes the treatment in RFC5661 in two major respects:

   - It allows extensions within minor versions
   - It deals with and creates rules for dealing with protocol changes that
   do not involve XDR changes.

> 5.2: COMPOUUNDS->COMPOUNDS

Will fix.

> think it's out of scope here and we should just remove the uses.

Will fix.

> been stated elsewhere.

This is the result to boil down the existing rule saying you could not
transfer any id of any sort (filehandle, stateid) across minor versions.

I found that rule hard to maintain in a situation in which minor versions
are guaranteed not to be enormous and in which you can add extensions.
As a result I thought I could maintain the original intention which was to
avoid chaos while not unduly restricting upgrade and also maintaining that
you could not add or remove extensions within the scope of a clientid.

> Is there an erratum?

No.

> 8.2: "to provides"->"to provide", "by a searching"->"by searching".

Will fix.

> client would hit, if I'm understanding right.

The motivation is to allow a server which supports version X+1
to support X as well and respond appropriately in terms of its
(lack of) knowledge of protocol elements should not know about,
even though they are part of the XDR it is using.

Clients that encountered these would be buggy but that doesn't
mean that the behavior specified should not be required.  if they
were required for interoperablity, the "shoulds would become
SHOULDs.

> 9: "the working will"->"the working group will"

Will fix.

> "the indexing material": I think you're referring to some stuff that got

True.  Will fix.

> I'm not sure section 9 is entirely necessary.

I don't think it is entirely necessary but I still think it is a good idea.

> I think this was
> or 4.0?

Actually, I put this stuff in originally at Tom Haynes request.  He was
concerned about
our inability to correct protocol errors that involved XDR changes  I don't
think I
should remove this unless Tom is OK with the change.

> I'd rather deal with these concerns when we do that.

My feeling is that it is better to deal with this kind of thing before you
get into
crisis mode.  My impression was that Tom's feelings were similar.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 18 Aug 2016 14:16:01 -0400

On Thu, Aug 18, 2016 at 12:52:41PM -0400, David Noveck wrote:
> If we did that, we would invalidate existing behavior,

There are no existing NFSv4.3 implementations, so no existing behavior
to invalidate.  I must be missing your point.

> > I understand the unknown vs. unsupported distinction, but I don't
> > really understand what benefit we get from it.  How would a client,
> > for example, behave differently if it discovered a server didn't
> > know of a certain extension, vs. whether that server didn't support
> > it?
> 
> What is different is what it is entitled to conclude about the the
> other elements of the protocol.  If a protocol element is known or
> unknown, the client is entitled to assume that all other protocol
> elements in the same feature package have are the same. On the other
> hand, if you find out that it is unsupported, you can't assume that.

Why not?

> For your particular extension, you may be able to identify these two
> concepts but the intention is to be able to do support more
> complicated extensions as well.

Maybe you could give a concrete example?

> you could not add or remove extensions within the scope of a clientid.

OK, I was missing an inter-version vs. intra-version distinction here,
thanks!

--b.

---

From: David Noveck <[email-masked]>
Date: Fri, 19 Aug 2016 09:34:40 -0400

> to invalidate.  I must be missing your point.

I guess the point is that while there may be no problems that arise
interacting
with an empty set of implementations, the value of doing so is quite
limited.

The original goal was to be able to easily add support to NFSv4.2 server
implementations and have:

   - All existing v4.2 client implementations operate without any change at
   all.
   - Make it very easy to add the new feature to v4.2 client so that they
   can use the new feature and have no difficulties with earlier servers
   implemented in the past.

Also, it should be easy to take v4,1 server implementations and recognize
the new v4.2 XDR and simply support none of the existing v4.2 OPTIONAL
features.  At that point they can implement extensions just as v4,2 servers
can.

In any case, we have a substantial set of:

   - Server implementations that can easily get support for the new feature.
   - Client implementation that can easily use server support for the new
   feature and interoperate correctly with servers that do not know about the
   new feature.

The goal is to have as large a set of implementations (client or server)
that are suitable to get the support, as people need it.  I don't think we
want to put obstacles in the path of that unless there is no choice.

> Why not?

The basic reason is that XDR recognition of a construct is not necessarily
tied to
implementation of that feature.  For specific examples, see below.

> Maybe you could give a concrete example?

Gee.  I thought you'd never ask :-)

As an example, consider draft-ietf-nfsv4-xattrs.  If you do a xattr-related
request, it
might not be supported on the particular file system you tried it on for a
number of
reason, including the possibility that the physical file system does not
support
xattrs.  As a result, you cannot conclude anything about xattr support on
other
file systems on the same server, based on whether it is supported on the
first
file system you try it on.

On the other hand, if the server implementation treats that operation as
unknown, you
may conclude that it has no support at all for that operation on any file
system and that
the same holds for any construct in the same feature package.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 19 Aug 2016 13:20:19 -0400

> features.  At that point they can implement extensions just as v4,2 servers

Hm, I don't know, this makes me slightly more nervous.  But can I think
of anythng this actually breaks?

I might worry that, for example, some user may have already learned that
if security label security support is causing them some kind of headache
then they can turn off 4.2 and be done with it.

I mean, ideally a) it wouldn't cause any headaches, and b) if it did,
we'd have other ways to disable it.  And I don't actually know of such a
case.  And if an implementation did find itself in such a case, it can
still choose not to do this sort of 4.2+-to-4.1 backport.

> want to put obstacles in the path of that unless there is no choice.

Hm.

The actual under-the-cover implementation of a new minorversion
shouldn't be much of an obstacle at this point.

Thinking about Linux implementation, I guess the difference is that
we've stuck the minor version number in the user interfaces, so for
example there may be a bunch of clients out there that now have
"version=4.1" stuck in a line in their fstab or in /etc/nfsmount.conf.
Those clients are probably still getting regular kernel updates, so we
could sneak upgrades in without requiring adminstrators to do anything
else.  Hm.  I'm saying "hm" a lot.  Is this rope useful, or will I just
hang myself on it?

You might convince me.  I don't feel that strongly, anyway, so OK, any
objection withdrawn.

> As an example, consider draft-ietf-nfsv4-xattrs.  If you do a
> xattr-related request, it might not be supported on the particular
> file system you tried it on for a number of reason, including the
> possibility that the physical file system does not support xattrs.  As
> a result,

(in the case that an operation returns NOTSUPP)

> you cannot conclude anything about xattr support on other file systems
> on the same server, based on whether it is supported on the first file
> system you try it on.
>
> On the other hand, if the server implementation treats that operation
> as unknown, you may conclude that it has no support at all for that
> operation on any file system and that the same holds for any construct
> in the same feature package.

Got it, so we can skip probing new filesystems if we get OP_ILLEGAL or
an xdr error.  To me, that's a very minor optimization that client
implementors could probably figure out on their own--but, OK.

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 22 Aug 2016 11:26:04 -0400

> > features.  At that point they can implement extensions just as v4,2 servers

Attributes may be interesting.

In theory the bitmasks are arbitrary length, in practice I seem to
recall clients having some bugs here (see e.g. e5012d1f3861 "NFSv4.1:
update nfs4_fattr_bitmap_maxsz" for the linux client).  I don't know how
many buggy clients are still out there.  But that would be something to
watch for if we tried to backport the umask thing to v4.0, as that would
mean adding a 3rd word.

We can recommend servers avoid the third word when possible.  (So,
prefer returning a two-word mask over returning a 3-word mask with last
word 0.)  That won't help with the supported_attributes attribute.

--b.