Subject: [nfsv4] Review of draft-ietf-nfsv4-rfc5666bis-02

From: David Noveck <[email-masked]>
Date: Wed, 13 Jan 2016 17:27:22 -0500

*General Comments*

*Overall Evaluation*

This version of the document no longer has placeholder sections.  I have
 focused this review on section 8 and 9, which were recently updated.  The
rest of the document is in pretty good shape.

This document has been progressing nicely but is not quite ready for
Working Group Last Call.  Unfortunately, there is no such thing as Working
Group Penultimate Call but I'm worried that if people wait until WGLC to
actually review this document, then we my have an unduly long last call.

*Review the Fine Draft (RTFD)*

There hasn't been a lot of discussion of this draft on the list.  I know
people are busy but this draft, with the clarifications already made is
well worth reviewing.  It is a lot easier to deal with than RFC5666 and if
it isn't sufficiently clear, the right time to point that out is now.  My
view is that, if you are at all concerned about performance, and I expect
most people are, RDMA  is in your future and it makes sense learn about it
now.

If your primary interest is in later versions of RDMA support, it makes
sense to look at section 9, and make sure that what is there is an
appropriate base for later development/enhancement.

*Comments by Section*

*5.2.  Fixed Header Fields*

Note the corresponding change in *9.1.  RPC-over-RDMA Version Numbering.*

Suggest replacing the last sentence by "These four fields, which must
remain with the same meanings  and in the same positions in all subsequent
versions of RPC/RDMA, are described in the subsections below."

*5.2.2 Version Number*

Suggest replacing the last sentence by "The rules regarding changes to the
version number are to be found in Section 9.1 as part of a full discussion
of protocol extensibility beginning in Section 9."

*5.3.  Chunk Lists*

In the first sentence, suggest deleting the "MUST".   We have an XDR
description and I don't think we should single out particular fields to get
a "MUST" as it might somehow be interpreted as suggesting that those that
don't have the "MUST" are in some way optional.

*5.3.2.  Write List*

In the first sentence of the second paragraph, suggest replacing
"corresponding data" by "data corresponding to a DDP-eligible XDR item".

In the last sentence of the second paragraph, suggest replacing "has" by
"can have".

Suggest adding the following sentence to the third paragraph:

In each case in which a DDP-eligible result is transferred via RDMA Write,
the associated data is removed from the XDR stream associated with the
reply.

Suggest adding the following new paragraph after the current third
paragraph.

It is the responsibility of the requester to associate a memory area with
each write chunk which is large enough to accommodate the longest possible
corresponding DDP-eligible XDR item.  In situations in which there could be
 multiple DDP-eligible XDR items in a reply, the requester may be unable to
predict which DDP-eligible XDR item a given chunk will be associated with.
In such cases, the requester needs to associate a memory area large enough
to accommodate the longest of the DDP-eligible XDR items that could
possibly be associated with the particular chunk.

*5.3.3.  Reply Chunk*

I don't see the logic behind the third paragraph.  In case in which a reply
chunk is provided because a reply might be larger than the inline threshold
but is in fact smaller, it seems that it is right for the responder to do a
send and not use the reply chunk since this is simpler and lower-overhead.
He always has to do a send but there is no reason to tell him to do an RDMA
Write in addition.

*5.4.  Memory Registration*

The second paragraph makes a good point.  Unfortunately, it seems to be
contradicted by what is in *5.4.1.  Registration Longevity*.  This whole
area needs to be clarified.

*5.5.2.  XDR Errors*

There's a problem with the interaction of this and Section 5.5.1.  Suggest
 replacing the last sentence of the first paragraph by the following:

In such cases, the responder MUST reply with an RDMA_ERROR message.  The
rdma_err value value MUST be ERR_CHUNK unless there is a version mismatch,
in which case the rdma_err value value ERR_VERS  MUST be returned as
described in Section 5.5.1.

*8.  Upper Layer Binding Specifications*

Typically RFC2119 terms are directed to a client or server implementer.
The existing first sentence seems to direct its "MUST" to a protocol or an
anonymous ULB author.  Suggest the following replacement:

To assure interoperable operation, each RPC program and version tuple that
operates on an RDMA transport needs to have an Upper Layer Binding
Specification. Clients MUST NOT use RPC/RDMA to make requests for protocols
that do not have a ULB. Similarly, servers MUST NOT respond to RPC/RDMA
requests for protocols that do not have a ULB.

In the second bullet:

   -

   Suggest replacing "on how to compute" by "regarding the determination of the"

   -

   Suggest replacing "operations" by "requests".  The term
"operations" can be confusing in the context of NFSv4 which only has a
single operation (i.e. COMPOUND) other than NULL.

*8.1.  DDP-Eligibility*

Part of the problem with writing this section is the asymmetry between
requests and replies:

   - For requests, if an item is sometimes appropriate for DDP and
   sometimes not, the sender can take advantage of the "MAY" and do what makes
   sense for that particular item.
   - For replies, the requester has to assign the write chunk, when doing
   the request, depriving him of the option to do what makes sense in the
   particular case.

It may eventually make sense to split these up into subsections but my
comments below are based on the assumption that that will not be done.

In the second paragraph, suggest replacing "is a candidate for eligibility"
by "should be designated as eligible".

With regard to item #1, "frequently much larger than the inline threshold"
may be too restrictive, particularly if the inline size threshold goes much
above 1K.  Suggest "often of substantial size" as a potential replacement.

In the first sentence of the the third (non-indented) paragraph, suggest
replacing "non-DDP-eligible" by "DDP-ineligible".

*8.1.1.  Write List Ordering Ambiguity*

The third paragraph, while saying that ambiguities can arise, is not clear
about how they can arise.  I don't believe any true ambiguities can arise
given that actual DDP-eligible XDR items are matched sequentially with
write chunks.

I think what is really being referred  to as an "ambiguity" are situations
in which the requester cannot predict, at the time of request issuing which
write chunk will match up with which potential DDP-eligible XDR  items.
Once the reply is generated, and it is known which DDP-eligible XDR items
actually exist, there is no ambiguity.  The only possible ambiguity is
already foreclosed by the last paragraph of 8.1.

*8.1.2.  DDP-Eligibility Violation*

I don't see that the second item in the first sentence ("or provides a
Write list when there are no DDP-eligible items allowed in the operation's
reply') is a DDP-eligibility violation. The idea is you have to at least as
many chunks as items but can have more. Having one chunk when there are
zero eligible items is in line with that. It is just not workable to
require the requester to know in advance how many of the chunks will in
fact be used. For example, if you do a COMPOUND that contains a READ and
there is an error that occurs before the READ, the requester will assign a
write chunk because he has to assume the READ might might succeed, but
there may be no DDP-eligible XDR item in the result.

I suggest deleting the third paragraph.

With regard to the second paragraph, I don't think "might attempt to parse
... anyway" is correct. He might do almost anything but I think it should
say he "MUST NOT" do that, since this would give rise to non-interoperable
behavior. If there are servers which do this now, we could weaken it to
"SHOULD NOT".

The basic problem with the third paragraph is that it is not clear how the
requester can determine that the third case has occurred. All it sees is
that there is a write chunk and a DDP-eligible XDR-item. There is no way
for him to reliably determine whether it has been reduced or not. If it had
access to the original XDR stream it could see whether the desired
reduction has occurred, but the reality is that it sees some bytes of the
XDR stream and all it can do, as far as I can see, is simply assume that
the specified reduction has occurred and interpret the XDR stream on that
basis. A possible replacement would be as follows:

Whenever a the requester, in decoding the XDR stream corresponding to a
reply reaches the point at which a DDP-eligible XDR item is expected and
there is a corresponding write chunk, it needs to assume that the XDR item
has been removed from the XDR stream and continue  the XDR decode on that
basis.  If the responder has erroneously not used the write chunk, and left
the DDP-eligible item in the XDR stream, the likely result is that an XDR
decode result would result which should be reported as described in Section
5.5.2.

*8.2.  Maximum Reply Size*

I have problems, with the third paragraph involving use of the words
"operations" (already discussed) and the term "mechanism" which gives the
wrong impression.  Suggest the following replacement:

If there are requests expressible in the upper Layer Protocol for which
there is no clear reply size maximum, it is the responsibility of the Upper
layer binding to provide a means by which a dependable upper bound for the
reply size can be determined..

*8.3.  Additional Considerations*

In the first bullet, should "recommend" be replaced by "RECOMMEND"?

*9. Transport Protocol Extensibility *If the first sentence were true,
there would be no need for RFCs 7530 and 5661 to be standards-track
documents and RFCs 7531 and 5662 would suffice to solely define the
protocols in question.
Suggest the following possible alternatives to deal with the first sentence:

   - replace the sentence by "The formats of messages within Upper Layer
   RPC Protocols are defined by the associated XDR definition."
   - delete the word "solely"

*9.1.  RPC-over-RDMA Version Numbering*

There are a number of problems with the first paragraph:

   - Adding the "start of the chunk lists" to the list of things that must
   always be kept the same is wrong, since the chunk lists are in a switched
   union in which some message type do not have chunk lists.  Also, there is
   no point in requiring new message types to always have the same set of
   chunk lists and require them not to put anything before the chunk lists.
   - While it makes sense to require that the first four fields always stay
   in the place same place, the reason given only explains why the version and
   message type need to be.  The reason to keep the others in a fixed place is
   "because the standard has decided so" and it is confusing to pretend
   otherwise.

Suggest the following replacement:

The first four fields MUST remain aligned at the same fixed offsets
for all versions of the RPC-over-RDMA header.  This aproaach has been
adopted because:

   - The version number must be in a fixed place in order allow
version mismatches to be detected.
   - For version mismatches to be reported in a fashion that all
future version implementations can reliably detect, the message opcode
must be in a fixed place, and the the error number and the response
indicating a version mismatch must always remain the the same.

With regard to the three bulleted conditions, I would have no problem with
the first one, if stated alone, but given the assemblage of conditions that
are stated, I'm not sure that the implications of the extensibility model
have been full considered. I'll deal with each condition in turn:

   1. As stated, this allows optional extensions to be added, as in the
   original conception of NFSv4 minor versioning. Just as you can add new
   operations to a compound, you can add new header types and the sender can
   find out if the receiver has support for the operation in question. I think
   that's fine but the question that needs to be addressed is what exactly is
   the function of the version number. Functionally it is has the same sort of
   role as the RPC program version, i.e. the *major version.* If that were
   the model, it would be a toss-up whether NFSv5 or RPC/RDMA version 2 would
   happen first. I have heard enough discussion about a potential Version Two
   to think there is some sort of disconnect here, that we need to address. It
   may be that people are thinking that somehow the version number is somehow
   on the level of the NFSv4 minor version. If so, it is like the change from
   minor version zero to minor version one, i.,e on the order of a minor
   version group change (as defined in draft-ietf-nfsv4-versioning). In any
   case, I'm really not clear what the intention is here, although the text
   is, as written, clear. If Chuck were to say , "Yes, I really mean that",
   I'd be Ok.
   2. First of all, I'm not sure why this here and why if you had an
   arrangement in which optional remote invalidation were provided for
   according to 1, Why would that require a version number change? Is that the
   intention? If it isn't, how does one know? Similarly, would use of private
   data during the RDMA/CM connect message exchange.require a version number
   change? I would hope not but, even if the answer is the one I would not
   choose, we need to be clear about exactly what this means.
   3. I'm also not sure why this is here. I don't see any likelihood that
   the set of transfer modes is changed, in a compatible way (item #1), I
   don't see why the version number should change in this case. Another issue
   the use of the word "allowable". Although there were no implementations of
   read-read, it was "allowable" in that someone could have implemented one in
   the context of RFC5666. On the other hand, in RFC5666bis it is not
   allowable. This could reasonably be interpreted as a change in the set of
   allowable transfer models requiring a version number change, but I
think we *really
   *don't want to go there. :-(

---

From: Chuck Lever <[email-masked]>
Date: Fri, 15 Jan 2016 11:27:33 -0500

Dave-

Sorry for the delay. I wanted to digest all of your comments
carefully before responding. I've addressed most of your
remarks with changes to the document. The changes will appear
in -03.

There are a few areas which merit further discussion.
Responses inline below.

> On Jan 13, 2016, at 5:27 PM, David Noveck <[email-masked]> wrote:
> 
> General Comments

[ ... snipped ... ]

> 5.3.2.  Write List
> 
> In the first sentence of the second paragraph, suggest replacing "corresponding data" by "data corresponding to a DDP-eligible XDR item".
> 
> In the last sentence of the second paragraph, suggest replacing "has" by "can have".
> 
> In each case in which a DDP-eligible result is transferred via RDMA Write, the associated data is removed from the XDR stream associated with the reply.
> 
> Suggest adding the following new paragraph after the current third paragraph.
> 
> It is the responsibility of the requester to associate a memory area with each write chunk which is large enough to accommodate the longest possible corresponding DDP-eligible XDR item.  In situations in which there could be  multiple DDP-eligible XDR items in a reply, the requester may be unable to predict which DDP-eligible XDR item a given chunk will be associated with.  In such cases, the requester needs to associate a memory area large enough to accommodate the longest of the DDP-eligible XDR items that could possibly be associated with the particular chunk.

[ ... snipped ... ]

> 8.1.  DDP-Eligibility
> 
> Part of the problem with writing this section is the asymmetry between requests and replies:
> 	• For requests, if an item is sometimes appropriate for DDP and sometimes not, the sender can take advantage of the "MAY" and do what makes sense for that particular item.
> 	• For replies, the requester has to assign the write chunk, when doing the request, depriving him of the option to do what makes sense in the particular case.
> It may eventually make sense to split these up into subsections but my comments below are based on the assumption that that will not be done.

It is true that with the Read-Write transfer model, a
responder has less flexibility than the requester.
Currently I believe this section can be made generic
enough to cover both the Read list and Write list.

But I think we have something missing in the earlier
Write List section (5.3.2). I can't find any kind of
guidance in RFC5666 regarding how many Write chunks a
requester is required to provide when a reply contains
DDP-eligible data items. Zero or more? At least one?
One for the maximum number of DDP-eligible data items
that can be returned? Are there any implementations
that do not provide a Write list when there is a
DDP-eligible data item in a reply?

Certainly the requester MUST provide a Reply chunk if
it doesn't provide a Write list and the DDP-eligible
data item can cause the reply size to exceed the
inline threshold.

> In the second paragraph, suggest replacing "is a candidate for eligibility" by "should be designated as eligible".
> 
> With regard to item #1, "frequently much larger than the inline threshold" may be too restrictive, particularly if the inline size threshold goes much above 1K.  Suggest "often of substantial size" as a potential replacement.

I've changed this bullet to read
 
o  The XDR data item is frequently sent or received,
    and its size is often much larger than typical
    inline thresholds.

I'm not sure I understand how basing this criteria
on inline thresholds restricts flexibility. Senders
already use the inline threshold to determine when
to use chunks. And if a data item that is not DDP
eligible makes an RPC message larger than the
receiver's inline threshold, a Long message can
always be used.

> In the first sentence of the the third (non-indented) paragraph, suggest replacing "non-DDP-eligible" by "DDP-ineligible".

[ ... snipped ... ]

> 9.1.  RPC-over-RDMA Version Numbering
> 
> 	• Adding the "start of the chunk lists" to the list of things that must always be kept the same is wrong, since the chunk lists are in a switched union in which some message type do not have chunk lists.  Also, there is no point in requiring new message types to always have the same set of chunk lists and require them not to put anything before the chunk lists.
> 	• While it makes sense to require that the first four fields always stay in the place same place, the reason given only explains why the version and message type need to be.  The reason to keep the others in a fixed place is "because the standard has decided so" and it is confusing to pretend otherwise. 
> The first four fields MUST remain aligned at the same fixed offsets for all versions of the RPC-over-RDMA header.  This aproaach has been adopted because:
> 	• The version number must be in a fixed place in order allow version mismatches to be detected.
> 	• For version mismatches to be reported in a fashion that all future version implementations can reliably detect, the message opcode must be in a fixed place, and the the error number and the response indicating a version mismatch must always remain the the same.
> With regard to the three bulleted conditions, I would have no problem with the first one, if stated alone, but given the assemblage of conditions that are stated, I'm not sure that the implications of the extensibility model have been full considered.  I'll deal with each condition in turn:
> 	• As stated, this allows optional extensions to be added, as in the original conception of NFSv4 minor versioning.  Just as you can add new operations to a compound, you can add new header types and the sender can find out if the receiver has support for the operation in question.   I think that's fine but the question that needs to be addressed is what exactly is the function of the version number.  Functionally it is has the same sort of role as the RPC program version, i.e. the major version.  If that were the model, it would be a toss-up whether NFSv5 or RPC/RDMA version 2 would happen first.  I have heard enough discussion about a potential Version Two to think there is some sort of disconnect here, that we need to address.  It may be that people are thinking that somehow the version number is somehow on the level of the NFSv4 minor version.  If so, it is like the change from minor version zero to minor version one, i.,e on the order of a minor version group change (as defined in draft-ietf-nfsv4-versioning).  In any case, I'm really not clear what the intention is here, although the text is, as written, clear.  If Chuck were to say , "Yes, I really mean that", I'd be Ok.

Perhaps we can postpone this part of the discussion
until I have presented a problem statement for a Version
Two. Then we will have concrete examples with which to
test our thinking.

> 	• First of all, I'm not sure why this here and why if you had an arrangement in which optional remote invalidation were provided for according to 1, Why would that require a version number change?  Is that the intention?  If it isn't, how does one know?  Similarly, would use of private data during the RDMA/CM connect message exchange.require a version number change? I would hope not but, even if the answer is the one I would not choose, we need to be clear about exactly what this means.

I believe we need to draw some line on the addition or
removal of abstract RDMA operations. I've changed
this bullet to read:

o  Whenever the use of a new abstract RDMA operation is
    required, or the use of an existing abstract RDMA
    operation is deprecated

My thinking for remote invalidation was that it would
not be part of Version One, even though we can do some
limited prototyping using current Version One
implementations. The use of "is required" above should
provide some leeway.

> 	• I'm also not sure why this is here.  I don't see any likelihood that the set of transfer modes is changed, in a compatible way (item #1), I don't see why the version number should change in this case.  Another issue the use of the word "allowable".  Although there were no implementations of read-read, it was "allowable" in that someone could have implemented one in the context of RFC5666.  On the other hand, in RFC5666bis it is not allowable.  This could reasonably be interpreted as a change in the set of allowable transfer models requiring a version number change, but I think we really don't want to go there. :-(

Transfer model is mentioned because it is plausible
that such support can be added without introducing
incompatible changes to the XDR. Note how little had
to change when Read-Read was removed from Version
One.

This bullet now reads:
 
o  Whenever a transfer model that is supported by
    current implementations is removed, or a transfer
    model that is not supported by current
    implementations is added

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 15 Jan 2016 13:37:27 -0500

> enough to cover both the Read list and Write list.

I think that will be possible.  I'd rate it as a 55-60% probability.
We'll see how it works out.

> Write List section (5.3.2).

I don't think anything is missing here.

> I can't find any kind of
> DDP-eligible data items.

I don't think any valid non-trivial guidance can be provided, in part
because of the write chunk side of
the asymmetry which we  were just discussing.

> Zero or more? At least one?
> that can be returned?

My answer would be "at least zero" :-)

> Are there any implementations

I don't know but I can give you an example where I believe a sensible
implementation would provide an empty write list.

Suppose you are doing a READ of a small number of byte (e.g. four).  In
that case, I would hope that the client would not post a write chunk and so
the responder would send the four data bytes right after the four bytes of
length in a inline send message.  In that case, registering the memory,
doing the explicit RDMA operation and unregistering the memory would be a
whole lot slower that doing the inline send.  I don't think the spec should
go out of its way to prevent that.

 >> 9.1.  RPC-over-RDMA Version Numbering

First of all, I'm assuming that we are in agreement about the comments  for
section 9.1 before we get to the three conditions for version number change.

If so, we can go on to discuss the version number change issue.

> Two. Then we will have concrete examples with which to

I don't think we should do this for a number of reasons:

   - I think there is a need to go forward on Version One as soon as we
   can.  Given that, it doesn't seem helpful to make a decision on the
   extension model for 5666bis dependent on a Version Two document that is yet
   to be written.
   - As the question at issue is really what is acceptable in (i.e. as a
   compatible extension to) Version One, a set of proposal/ideas for Version
   Two, is one thing we certainly don't want to wait for.
   - I think I have mentioned already (i.e below) some concrete examples of
   possible compatible extensions to Version One.

>       • First of all, I'm not sure why this here and why if you had an
according to 1,
>         Why would that require a version number change?  Is that the
intention?  If it isn't, how does one know?  Similarly, would use of
private data
>         during the RDMA/CM connect message exchange.require a version
number change? I would hope not but, even if the answer is the one I would
not choose, we need to be clear about exactly what this means.

> removal of abstract RDMA operations.

I think the appropriate line to draw is the one you draw in condition #1,
i.e. it it is a required use leading to incompatibility, then it requires a
version number change.

> I've changed this bullet to read:

>     required, or the use of an existing abstract RDMA

I can live with this.

> not be part of Version One

I was thinking it would be specifiable as a compatible extension, that
would not require changing the

> even though we can do some
> implementations.

My thinking would be that once appropriate prototyping is done
(successfully), we would move the appropriate
document forward.

If, as appears to be the case, this would not require a version number
change, this would be within the scope of
Version One, even it was not specified in rfc5666bis.

To borrow some terminology that Mike Kupfer and I have arrived at (in the
context of draft-ietf-nfsv4-versioning),
it would then be part of the current Version One even though not part of
the base Version One.

> The use of "is required" above should
> provide some leeway.

I think it provides sufficient leeway although "specified as REQUIRED" would be
preferable.

>>       • I'm also not sure why this is here.
            I don't see any likelihood that the set of transfer modes is
changed, in a compatible way (item #1),
            I don't see why the version number should change in this case.
Another issue the use of the word "allowable".
           Although there were no implementations of read-read, it was
"allowable" in that someone could have implemented one
           in the context of RFC5666.  On the other hand, in RFC5666bis it
is not allowable.  This could reasonably be interpreted
           as a change in the set of allowable transfer models requiring a
version number change, but I think we really don't want to go there. :-(

> to change when Read-Read was removed from Version

In particular the version number didn't have to change and, if condition #3
wasn't there, it wouldn't have to change either.

>     model that is not supported by current

I can live with this, but I'd prefer it if it were removed.

On Fri, Jan 15, 2016 at 11:27 AM, Chuck Lever <[email-masked]>
wrote:

> > In each case in which a DDP-eligible result is transferred via RDMA
> Write, the associated data is removed from the XDR stream associated with
> the reply.
> >
> > It is the responsibility of the requester to associate a memory area
> with each write chunk which is large enough to accommodate the longest
> possible corresponding DDP-eligible XDR item.  In situations in which there
> could be  multiple DDP-eligible XDR items in a reply, the requester may be
> unable to predict which DDP-eligible XDR item a given chunk will be
> associated with.  In such cases, the requester needs to associate a memory
> area large enough to accommodate the longest of the DDP-eligible XDR items
> that could possibly be associated with the particular chunk.
>
> [ ... snipped ... ]
>
> > 8.1.  DDP-Eligibility
> >
> >       • For requests, if an item is sometimes appropriate for DDP and
> >       • For replies, the requester has to assign the write chunk, when
> doing the request, depriving him of the option to do what makes sense in
> the particular case.
> > In the second paragraph, suggest replacing "is a candidate for
> eligibility" by "should be designated as eligible".
> >
> > With regard to item #1, "frequently much larger than the inline
> threshold" may be too restrictive, particularly if the inline size
> threshold goes much above 1K.  Suggest "often of substantial size" as a
> potential replacement.
>
> >       • Adding the "start of the chunk lists" to the list of things that
> must always be kept the same is wrong, since the chunk lists are in a
> switched union in which some message type do not have chunk lists.  Also,
> there is no point in requiring new message types to always have the same
> set of chunk lists and require them not to put anything before the chunk
> lists.
> >       • While it makes sense to require that the first four fields
> always stay in the place same place, the reason given only explains why the
> version and message type need to be.  The reason to keep the others in a
> fixed place is "because the standard has decided so" and it is confusing to
> pretend otherwise.
> > The first four fields MUST remain aligned at the same fixed offsets for
> all versions of the RPC-over-RDMA header.  This aproaach has been adopted
> because:
> >       • For version mismatches to be reported in a fashion that all
> future version implementations can reliably detect, the message opcode must
> be in a fixed place, and the the error number and the response indicating a
> version mismatch must always remain the the same.
> > With regard to the three bulleted conditions, I would have no problem
> with the first one, if stated alone, but given the assemblage of conditions
> that are stated, I'm not sure that the implications of the extensibility
> model have been full considered.  I'll deal with each condition in turn:
> >       • As stated, this allows optional extensions to be added, as in
> the original conception of NFSv4 minor versioning.  Just as you can add new
> find out if the receiver has support for the operation in question.   I
> think that's fine but the question that needs to be addressed is what
> exactly is the function of the version number.  Functionally it is has the
> same sort of role as the RPC program version, i.e. the major version.  If
> that were the model, it would be a toss-up whether NFSv5 or RPC/RDMA
> version 2 would happen first.  I have heard enough discussion about a
> potential Version Two to think there is some sort of disconnect here, that
> we need to address.  It may be that people are thinking that somehow the
> version number is somehow on the level of the NFSv4 minor version.  If so,
> it is like the change from minor version zero to minor version one, i.,e on
> the order of a minor version group change (as defined in
> draft-ietf-nfsv4-versioning).  In any case, I'm really not clear what the
> intention is here, although the text is, as written, clear.  If Chuck were
> to say , "Yes, I really mean that", I'd be Ok.
>
> >       • First of all, I'm not sure why this here and why if you had an
> according to 1, Why would that require a version number change?  Is that
> the intention?  If it isn't, how does one know?  Similarly, would use of
> private data during the RDMA/CM connect message exchange.require a version
> number change? I would hope not but, even if the answer is the one I would
> not choose, we need to be clear about exactly what this means.
>
> >       • I'm also not sure why this is here.  I don't see any likelihood
> that the set of transfer modes is changed, in a compatible way (item #1), I
> allowable transfer models requiring a version number change, but I think we
> really don't want to go there. :-(
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 15 Jan 2016 15:17:53 -0500

> On Jan 15, 2016, at 1:37 PM, David Noveck <[email-masked]> wrote:
> 
> >       • First of all, I'm not sure why this here and why if you had an arrangement in which optional remote invalidation were provided for according to 1, 
> >         Why would that require a version number change?  Is that the intention?  If it isn't, how does one know?  Similarly, would use of private data 
> >         during the RDMA/CM connect message exchange.require a version number change? I would hope not but, even if the answer is the one I would not choose, we need to be clear about exactly what this means.
> 
> I think the appropriate line to draw is the one you draw in condition #1, i.e. it it is a required use leading to incompatibility, then it requires a version number change.
> 
> I was thinking it would be specifiable as a compatible extension, that would not require changing the 
> 
> My thinking would be that once appropriate prototyping is done (successfully), we would move the appropriate 
> document forward.
> 
> If, as appears to be the case, this would not require a version number change, this would be within the scope of
> To borrow some terminology that Mike Kupfer and I have arrived at (in the context of draft-ietf-nfsv4-versioning),
> it would then be part of the current Version One even though not part of the base Version One.
> 
> I think it provides sufficient leeway although "specified as REQUIRED" would be preferable.
> 
>             I don't see any likelihood that the set of transfer modes is changed, in a compatible way (item #1), 
>             I don't see why the version number should change in this case.  Another issue the use of the word "allowable".  
>            Although there were no implementations of read-read, it was "allowable" in that someone could have implemented one 
>            in the context of RFC5666.  On the other hand, in RFC5666bis it is not allowable.  This could reasonably be interpreted 
>            as a change in the set of allowable transfer models requiring a version number change, but I think we really don't want to go there. :-(
> 
> In particular the version number didn't have to change and, if condition #3 wasn't there, it wouldn't have to change either.
> 
> > This bullet now reads:
> 

Without the second and third bullets, I don't see that
this section has much value at all. Saying the version
number must be bumped when incompatible changes are made
to the protocol seems to state the obvious.

I'll consider removing Section 9 in the next revision.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 15 Jan 2016 18:40:23 -0500

> Without the second and third bullets, I don't see that this section has
much value at all.

Although my view is different, if you really think
these additional constraints are necessary to the RPC/RDMA
extensibility model, I'm willing to discuss these issues further.  To me,
it is important to have a clear extensibility model, and I don't
want disagreements on minor matters to interfere with that.

> I'll consider removing Section 9 in the next revision.

I think you should consider why you chose to add a section about
extensibility to rfc5666bis.

RFC5666 didn't have one and I think that the result was that it wasn't very
clear what the extensibility model was, so I think your decision to address
this issue was the correct one.

If asked about RFC5666 extensibility, most people might have agreed that
condition #1 was valid or even obvious.

The big problem was that a large number of people might also have thought
that any change to the XDR would require a version number change, and some
might even have considered that obvious.

With your -02,that is no longer obvious but it might help to be more
explicit about the issue.

Also, the material now in the first paragraph of section 9.1 needs to be
retained no matter what you choose to do about the extensibility model.

On Fri, Jan 15, 2016 at 3:17 PM, Chuck Lever <[email-masked]> wrote:

> > I think the appropriate line to draw is the one you draw in condition
> #1, i.e. it it is a required use leading to incompatibility, then it
> requires a version number change.
> >
> > To borrow some terminology that Mike Kupfer and I have arrived at (in
> the context of draft-ietf-nfsv4-versioning),
> >             I don't see why the version number should change in this
> case.  Another issue the use of the word "allowable".
> >            in the context of RFC5666.  On the other hand, in RFC5666bis
> it is not allowable.  This could reasonably be interpreted
> >            as a change in the set of allowable transfer models requiring
> a version number change, but I think we really don't want to go there. :-(
> >
> > In particular the version number didn't have to change and, if condition
> #3 wasn't there, it wouldn't have to change either.
> >
> > > This bullet now reads:
> >

---

From: Chuck Lever <[email-masked]>
Date: Sun, 17 Jan 2016 14:46:01 -0500

> On Jan 15, 2016, at 6:40 PM, David Noveck <[email-masked]> wrote:
> 
> > Without the second and third bullets, I don't see that this section has much value at all.
> 
> Although my view is different, if you really think these additional constraints are necessary to the RPC/RDMA extensibility model, I'm willing to discuss these issues further.  To me, it is important to have a clear extensibility model, and I don't want disagreements on minor matters to interfere with that.
> 
> I think you should consider why you chose to add a section about extensibility to rfc5666bis.
> 
> RFC5666 didn't have one and I think that the result was that it wasn't very clear what the extensibility model was, so I think your decision to address this issue was the correct one.
> 
> If asked about RFC5666 extensibility, most people might have agreed that condition #1 was valid or even obvious.
> 
> The big problem was that a large number of people might also have thought that any change to the XDR would require a version number change, and some might even have considered that obvious.
> 
> With your -02,that is no longer obvious but it might help to be more explicit about the issue.

I consider it obvious that a version number bump is
required when incompatible changes are made to any
protocol. I added Section 9 because I thought there
would be more to say than the obvious. It's now not
clear to me what can be said with certainty while
there is only one protocol version in the books.

The discussion in Section 9 is really a set of
protocol design considerations, nothing to do with
guidance for implementers. Seems like a better
place for this type of language would be a separate
informative RFC?

Moreover, if we are planning to extend Version One
using only XDR extensibility practices, then a
Version Two is unnecessary, as you have pointed out.

By this reasoning, considering version number
changes in rfc5666bis is academic. It is only
valuable to consider if we actually need to change
the version number, which is beginning to appear
doubtful.

But note carefully -- the protocol error codes
ERR_VERS and ERR_CHUNK do not distinguish between
an illegal rdma_proc value and other XDR parsing
problems. I seem to recall that distinction was a
pre-requisite for effective XDR extensibility.

(RDMA CM private data does not have this problem,
but surely we don't want to solve _all_ of our
problems this way).

> Also, the material now in the first paragraph of section 9.1 needs to be retained no matter what you choose to do about the extensibility model.

I'm considering spelling out the reasons for
retaining the fixed header fields in future
versions possibly in Section 5.2.2 or 5.5. Even
that is stretching it for a protocol
specification. We can't say MUST, as you pointed
out, as these reasons are not instructions to
implementers.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sun, 17 Jan 2016 19:50:26 -0500

> place for this type of language would be a separate

If you have a protocol that has extensibility features defined within it,
such as a version number, how that is to be used should be clearly
specified in the document itself.

I think you need an extensibility section in rfc5666bis.  It should clearly
state that, unlike the rest of the spec,  it applies to all versions, and
that, for example,

   - New versions can be published as separate protocols without updating
   the RFC5666bis RFC.
   - Any change to the extensibility model will require updating the
   RFC5666bis RFC.

It is true that this is different from the case iof NFSv4 in which
versioning is defined in a separate document, but:

   - We only were forced to do this as a separate document because this
   wasn't done right in RFC3530.
   - That document is standards-track and I think for god reason.  In fact,
   one of the big problems was that RFC3530, despite being standards track,
   did not give clear guidance and said essentially, "OBTW, if the rules are
   a problem go ahead and change them".  The result was not good.   Making it
   informational would have given that same flawed message. I think we need
   a standards track document/section for this sort of thing.

An extensibility section will primarily be read by those defining
extensions.  I'd hope that those people would be implementers as well.  The
group of implementers who have no interest in defining extensions could
simply skip it.

> using only XDR extensibility practices,

I don't think we have any specific extension plans yet.

My view is that we need to do a lot with extensions, but we will probably
need/want a new version at some point.  I just want to make sure that if
we do a Version Two it will be for a good reason.

> then a Version Two is unnecessary, as you have pointed out.

I'm not sure exactly what I said on the subject, but I don;t believe that a
Version Two is necessary now, but I think it will be desirable at some
point and may be necessary at some later point.

> the version number, which is beginning to appear

I think you need to decide on an extensibility model, get the working
group's agreement, and stick to it.  When people have a good idea and want
to implement it is the worst time to address this.  That is my conclusion
from the v4.1 experience.

> pre-requisite for effective XDR extensibility.

That's exactly the sort of issue that needs to be addressed in an
extensibility section, and reviewed by the working group.

> but surely we don't want to solve _all_ of our

If it would, it would be great :-).  Unfortunately, that seems highly
unlikely :-(

A couple of notes about use of RDMA/CM private data:

   - Whatever is done in this area should have integral versioning support,
   separate from the RPC/RDMA version number.  Two possibilities are
   a separate version number associated with the private data format
   and a structure something like nfsv4 attributes in which the structure is a
   collection of elements each of which is present or not.
   - Having an area in the private data  with a bit mask for supported
   optional features (or known rdma_proc values) could simplify parsing and
   feature discovery for optional features.

> out, as these reasons are not instructions to

I think you shouldn't say "MUST" but I think you can and should say "must".

Here's a quote that relates to the issue from
draft-ietf-nfsv4-versioning-03:

   - Where this document does not directly specify implementation
requirements, use of these capitalized terms is often not appropriate,
since the guidance given in this document does not directly affect
interoperability.

   - In this document, what authors of RFCs defining features and
minor versions need to do is stated without these specialized terms.
Although it is necessary to follow this guidance to provide successful
NFSv4 version management, that sort of necessity is not of the sort
defined as applicable to the use of the keywords defined in [RFC2119].

The fact that these capitalized terms are not used should not be
interpreted as indicating that this guidance does not need to be
followed or is somehow not important.

Note that standards-track documents, such RFC2119 and
draft-ietf-nfsv4-versioning can and do say things directed to authors as
well as implementers. Note the following quote from RFC2119 which seems to
me to use "MUST" incorrectly:

In particular, they MUST only be used where it is actually required
for interoperation or to limit behavior which has potential for
causing harm (e.g., limiting retransmisssions)  For example, they must
not be used to try to impose a particular method on implementors where
the method is not required for interoperability.

Even thouh I think the "MUST" is incorrect, I believe the "must" is
appropriately directed to RFC authors.

On Sun, Jan 17, 2016 at 2:46 PM, Chuck Lever <[email-masked]> wrote:

> > Although my view is different, if you really think these additional
> constraints are necessary to the RPC/RDMA extensibility model, I'm willing
> to discuss these issues further.  To me, it is important to have a clear
> extensibility model, and I don't want disagreements on minor matters to
> interfere with that.
> >
> > RFC5666 didn't have one and I think that the result was that it wasn't
> very clear what the extensibility model was, so I think your decision to
> address this issue was the correct one.
> >
> > The big problem was that a large number of people might also have
> thought that any change to the XDR would require a version number change,
> and some might even have considered that obvious.
> >

---

From: Christoph Hellwig <[email-masked]>
Date: Mon, 18 Jan 2016 09:22:42 +0100

>    collection of elements each of which is present or not.

Version numbers for features are a bad idea.  Every local file system
has learned the hard way that feature bit mask of some sort are an
absolute must.

>    feature discovery for optional features.

I think this shoud be folded into the above bit mask.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 18 Jan 2016 09:51:46 -0500

> On Jan 17, 2016, at 7:50 PM, David Noveck <[email-masked]> wrote:
> 
> 	• Whatever is done in this area should have integral versioning support, separate from the RPC/RDMA version number.  Two possibilities are a separate version number associated with the private data format and a structure something like nfsv4 attributes in which the structure is a collection of elements each of which is present or not.

This could be done with something akin to TCP options.
The RDMA CM private data would contain a list or counted
array (similar to NFSv4 COMPOUND) of option elements.

Options could then be defined completely independently
of each other.

If a responder does not recognize an option in the
list, it simply does not put an element for that
option in the private data sent back to the
requester.

> 	• Having an area in the private data  with a bit mask for supported optional features (or known rdma_proc values) could simplify parsing and feature discovery for optional features.

An option that provides a range of valid rdma_proc values
(or a maximum valid rdma_proc value) would address the
missing ERR_ value I mentioned earlier.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 18 Jan 2016 10:24:35 -0500

Regarding "*folded into* the above bit mask" (emphasis added), let me
clarify my intentions and consider, in light of Christoph's suggestions,
some further possibilities for RDMA/CM private data origami.

My original proposal was for:

   - As a first possibility, there was a version number.  This version was
   not to describe a feature but was to describe the overall format of the
   private data.  It makes sense for this to be first word (or byte).
   - As an alternative to that, I proposed variable-length bitmask, in the
   style of NFSv4 attributes.  Each mask bit was to describe the presence or
   absence of some particular piece of data, that would follow the mask, with
   all present pieces of data following, in order of bit position.
   - There would be a mask of features in either case, which would be in at
   defined offset in the first case and controlled by a specified mask bit
   number in the second case.
   - In the second case, I also proposed, that there would be a bit mask
   position for a mask of message types supported by the sender.

I think it makes sense to combine these, as Christoph suggested, so that
each bit mask position either

   - Defines a bit mask position which specifies a piece of data but not an
   associated optional feature.   For example, the inline threshold.
   - Defines a bit mask position which specifies the presence of a
   particular optional feature with no associated data item.  For example,
   remote invalidation.
   - Defines a bit mask position which specifies the presence of a
   particular optional feature with an associated data item needed to support
   that optional feature.  One possible example of this would be support for
   and buffer configuration information relevant to use of  possible
   RDMA_MSG_DDP, a possible extension to replace the ill-fated RDMA_MSGP.

The big issue with regard to bit masks (other than the mask of message
types supported) is how the bit positions get assigned.  You want it to be
easy to get a bit position (to not impede development/prototyping) but want
to have some control to prevent waste/confusion.

Given that there will be very few optional features in real life, one
interesting approach is to allow a bit mask for well-settled assignments
 but supplement it with a list of easily gotten unique ids. to form an
extended feature/field designator.  It would consist of:

   - A bitmask of assigned bits, either assigned in an RFC or maintained by
   IANA on a standards-action basis.
   - A variable-length array of unique ids, each trivially obtained by
   developers of features.  My favorite format would combine a 48-bit MAC
   address, with a 48-bit count of milliseconds since RFC5666 was published.

So, to update my suggestion:

   - The first field would be a 32-bit version number.   I've never
   regretted putting in a version number (even if it turned out not to be
   needed) but I've often regretted not putting one in.
   - The second field would be a mask of message types supported by the
   sender (in the role of a receiver).
   - Following that would be an extended feature/field designator,
   combining designators for optional features and data fields as discussed
   above.
   - After that would be any associated data fields with those designated
   by the bit mask position appearing first.  FIelds designated by unique ids
   would follow in order of their position in the unique-id array.

---

From: William Allen Simpson <[email-masked]>
Date: Mon, 18 Jan 2016 13:36:44 -0500

Some thoughts:

1) Bit masks are a terrible idea.  Long-standing IETF design
principles have almost always used labels: either TLV option
numbers or descriptive strings.  We are not space constrained
on something that happens at most once per connection.

When the PPP WG wanted bit-masked sub-options, they always
devolved into something that would have been better handled
with multiple options.  In one particularly horrendous case,
Microsoft self-assigned the next available number, thinking it
was an option number -- when it was actually a bit mask of
supported options.  But we didn't find out about the problem
until they released Windows 95 shortly after bit value 4 was
assigned....

2) None of this is relevant to 5666bis.  It was not in 5666.
It will not be in 5666bis either, which consists of only
features implemented by multiple interoperable implementations.

3) What implementation are you doing?

---

From: Chuck Lever <[email-masked]>
Date: Mon, 18 Jan 2016 14:59:59 -0500

> On Jan 18, 2016, at 1:36 PM, William Allen Simpson <[email-masked]> wrote:
> 

It is relevant to rfc5666bis.

While the RDMA/CM private data format is to be specified
in a separate document, the authors of rfc5666bis need
to ensure that the language of rfc5666bis does not prevent
RPC-over-RDMA Version One from being extended in
reasonable ways.

--
Chuck Lever