Subject: [nfsv4] RPC/RDMA Stuff

From: David Noveck <[email-masked]>
Date: Mon, 19 Oct 2015 13:50:34 -0400

*Overall Comments*

*About Chuck's Document*

Chuck's document has convinced me that RFC5666 is worth saving (via an
rfc5666bis) and I urge everyone to read it.  If you haven't read RFC 5666
and care about RDMA as a possible way to improve networking performance,
read the first few sections of Chuck's document and keep the rest of the
document handy as a helpful as a guide to be pulled out when things are
unclear.

I'd like to thank Chuck for driving the RDMA effort and writing the
document and Tom Talpey for providing the necessary background to allow
RFC5666 to be rescued.

*Regarding the Need for rfc5666bis*

I think we need to reach a rough  consensus regarding the future of
RPC/RDMA.  Despite the strength of Chuck's arguments regarding the need to
do an rfc5666bis, we need to actually come to a decision about that and I
think we need to come to decision on this fairly soon.

As a way of beginning the process leading to that decision let me state my
suggestions about what I think we should decide:

   - As a way of encouraging working group involvement in this area, I
   suggest that we ask the working group members to review Chuck's document
   and then ask Chuck to incorporate the resulting input into a
   draft-ietf-rfc5666-implementation-experience-00.
   - At some point thereafter, if there is a working group consensus to
   proceed along that path, we should start work on a draft-ietf-rfc5666bis

*Regarding RPC/RDMA Version 2*

While I agree with Chuck that rfc5666bis should have priority, I think
discussion of Version Two should proceed and will be helpful, even though
we are not ready to make this a working group item.  I'm working on an I-D
with some suggestions and hope to get some time to make it publishable.

If we do proceed with a version 2, we should make sure it is a compatible
extension of Version 1.

*Comments on Chuck's Document by Section*

*This is a review of draft-cel-rfc5666-implementation-experience-01.  I had
gotten pretty far into it when the -02 was posted.*

*1.2 Purpose of this Document*

Regarding the last bullet item, it is not clear how one can have "protocol
enhancements that might be added without wire behavior changes".  If there
are no changes in behavior, the protocol is the same and not enhanced.

Maybe this could by changed to one of the following:

   - protocol enhancements that might be added without introducing
   incompatible wire behavior changes
   - protocol enhancements that might be added without changes to the
   existing Version One XDR
   - protocol enhancements that might be added while allowing existing
   implementations to successfully interoperate with those using the
   enhancements.

*1.3 Updating RFC 5666*

Regarding the first sentence, as there is no such things as an "official"
summary, saying it is "unofficial" is redundant.  Suggest replacing this
with "the author's".

With regard to the fourth (unnumbered) paragraph, I think there is a
fundamental difficulty that needs to be addressed, If RFC 5666 is
sufficiently unclear that multiple good-faith interpretations can result in
non-interoperable implementation, and such exist, then it isn't clear what
the protocol *is. *In other words, you can't maintain interoperability with
a set of implementations that are not interoperable among themselves.

I'd suggest something like the following:

There was general agreement at IETF 92 regarding the need to update
and obsolete
RFC 5666, while retaining the maximum possible degree of
interoperability with current RPC-over-RDMA Version One implementations.
This approach would avoid changes to on-the-wire behavior (number 3 above).
This would ease the burden on implementers, who could then reference a
single specification of the protocol. In addition, this alternative extends
the life of those current interoperable implementations in the field, which
have utilized RPC-over-RDMA Version One effectively.

With regard to the penultimate paragraph, I'm struck with the difference in
tone between this paragraph which suggests that issues with RFC 5666 "could
result in interoperability failure" and the second paragraph of section 1.2
which suggests that this is a "critical" issue.  I believe that section 1.2
is consistent with what Chuck intends and would suggest something on the
order of the following as a replacement:

Subsequent discussion within the nfsv4 Working Group has focused on
resolving specification ambiguities that make the production of
interoperable implementations unduly difficult. A Version Two of
RPC-over-RDMA, where deeper changes can be made and new functionality
introduced,remains a possibility to address with further discussion. Many
people feel and the author agrees that an important near-term priority is
addressing issues with the current Proposed Standard.

 I have the following issues with the last paragraph:

   - I'm not sure exactly what "accepted by the nfsv4 Working Group" means.
   - The suggestions in this document can be used as input regardless of
   any acceptance (Or not) by the working group, or whether we are doing
   5666bis or something else.
   - There is an assumption that a decision will be made to do 5666bis.
   While I think that is a good idea (see *Regarding the Need for *
   *rfc5666bis*) above, no such decision has been yet.  I hope it will be
   made soon.

I'd suggest replacing this by something like:

The recommendations in this document are directed to the preparation of an
rfc566bis.  Nevertheless, most of the observations and many of the
suggestions should prove useful regardless of the decision that
the working group makes with regard to the future of RPC/RDMA.

*1.4 Scope of this Document*

I don't see the point of this section.  The rest of the introduction
clearly states what this document is.  There is no reason for another
section saying the things it isn't.

*2. RPC-over-RDMA Essentials*

This section, including all sub-sections is big help.  A lot of it could go
into an eventual rfc566bis, which I think we should start on soon. (see
above).  Until then, anyone interested in RPC/RDMA would be well advised to
read this section before or soon after reading RFC 5666.

*2.2 Remote Direct Memory Access*

With regard to the first sentence of the third paragraph, I have two
comments:

   - It is not true to say the the CPU's are "not involved".  In fact the
   CPU on the client is involved in registering/unregistering memory and the
   CPU on the client is involved in initiating the RDMA transfer and
   responding to its completion.  This is a lot more efficient than copying
   but it is still not "no involvement."   As it turns out, much of this
   overhead is something that we will try to reduce in rfc5666bis if we can,
   and want to address in an RPC/RDMA Version Two.  It makes it harder to
   address this overhead if you don't recognize that it exists.
   - The degree of CPU involvement is smaller when the send/receive model
   is used.  When VIA-style interfaces are used, as they are for RNICs, the
   benefits derive from that interface concept and not from the use of RDMA
   per se.

It also should be noted that the rest of that paragraph applies to
send/receive as well as to RDMA.

While the fourth paragraph is a correct description on RFC5666, it should
be noted that how receive works is important in understanding the
protocol.  RFC5666 is not alone in this.  The Infiniband documentation has
seemingly endless detail regarding send semantics and almost nothing about
receive.

I don't think the introductory clause in the last paragraph is correct.
While it is true that RDMA is more efficient when used for larger
transfers, the same is true of send/receive.  The reason send/receive is
not used is that you have to post a read of the appropriate size to the
appropriate queue-pair in advance.  It may be that explaining the real
reason requires too much IB/VIA background and you can just say that these
classes are handled differently without offering an explanation.

*2.3 Transfer Models*

Bravo!  This section (and the 2.3.x sections) have helped clarify things
for me.

*2.4 Upper Layer Binding Specifications*

The forthright statement in the first paragraph is a big help.

It should be noted that RFC5666 seems quite reluctant to acknowledge the
role of binding specifications.  They are first mentioned in Section 9.

The abstract says:

The RDMA transport binding conveys the benefits of efficient,
bulk-data transport over high-speed networks, while providing for
minimal change to RPC applications and with no required revision of
the application RPC protocol, or the RPC protocol itself.

which makes the process seem much more turnkey than it in fact is. It may
be that mention of ULBs is obscured because it contradicts that narrative.

*2.5.3 Read Chunk*

I see some problems here:

   - First,, I found the use of the term "argument" confusing. While the
   data to be written by an NFSv3 WRITE is clearly an argument of the WRITE
   RPC, the situation is different in NFSv4. The buffer to be written by the
   WRITE operation is not an argument of the COMPOUND RPC. It is part of the
   argument but is not an argument itself.
   - The possibility that RPC object can be nested within one another,
   means that the same XDR position can be associated with multiple RPC
   objects that start at the same position and end in different positions.
   This possibility needs to be addressed in some way.
   - In cases in which you have a counted array, the XDR position of the
   counted array is the first byte of the length. on the other hand, for
   practical reasons, you want don't want to include the length in the area
   you RDMA. It has to be made clear how this issue is dealt with.

*2.5.4 Write Chunk*

It is not made clear here (or in 2.5.6) how the appropriate result/object
is selected, given that the write chunk doesn't have an XDR position. I
assume that successive list entries are assigned to successive
RDMA-eligible result objects as defined by the ULB, but that needs to be
stated explicitly. Also the ULB gets to decide where in the object the
RDMA-eligible piece is (i.e. it need to address the write-chunk version of
the counted array issue).

*2.5.5 Read List*

I assume that this list is in non-descending order of XDR position, but
that is not stated explicitly. If it is so, it should be stated explicitly
and if this is not required this should be noted to make the receiver aware
that he cannot assume it will be ordered.

*3.7 Write List Error Cases*

It isn't clear what sort of situation is being referred to in the
penultimate paragraph. If the ULB defines what is RDMA-eligible, I don't
see how the server can have any "choice" as to what to transfer via RDMA.

*5.1 Multiple RDMA-eligible Arguments an Results*

While there might be "additional considerations when marshaling or
decoding" COMPOUND's with multiple bulk items, I can't see any right now,
and I think that the fact Chuck, who has looked carefully at RFC5666,
hasn't found any, is a strong indication that, if they exist, they are
quite hard to find.  I don;'t think we should allow speculation about such
issues to hold up other needed work.

*5.1.1 Recommendations *

As regards the first paragraph, I can't disagree that this additional work
"may be needed" but I would express it as "may or may not be needed".

As regards the first sentence of the second paragraph, I don't understand
exactly what is proposed. I think the principle thing that needs to be done
is to review Chuck's document with an awareness that multiple chunks may be
present and I've tried to do that.

As regards the second sentence of the second paragraph, I don't understand
how ambiguity can arise. If it does arise, I'm not sure how this related to
having multiple bulk-data items in a COMPOUND.

*5.2 Bi-directional Operation*

With regard to the first sentence of the second paragraph, "RDMA's unique
requirements to pre-post receive resources", while unique to RDMA, are not
unique to the backwards direction.  They are the same requirements as for
the forward direction.

*5.2.1 Recommendations *

I don't see why this should be "Informational". NFSV4.1 is a
standards-track protocol which requires a backchannel. RPC/RDMA is
standards-track protocol which should support NFSv4.1. I'd prefer the
following replacement:

RFC 5666bis should provide support for backwards-direction RPC requests
adequate to support the needs of the NFSv4.1 backchannel.

*5.3 Missing NFS Binding Specifications*

Although the first sentence speaks of fully supporting NFSv4 minor
versions, the first two bulleted items have nothing to do with NFSv4. I
suggest they be deleted or moved somewhere else. Since these are unlikely
to ever be addressed, they should not be here to add confusion about the
need to fully support NFSv4.

As regard the last sentence of this paragraph, if this work needs to be
done, it would be better to say how it will be done (or say "I don't know"
if you don't) rather than simply saying how it won't be done.

With regard to the fourth existing bullet item, I don't see what would be
done here. The only data access protocols that are RPC-based use some
version of NFS. That may change in the future but I don't expect it to
change for a while.

With regard to the fifth existing bullet item, I've already partially
addressed this in draft-ietf-nfsv4-versioning-02. I think the eventual
NFSv4.x LUB will compatible with delegating the issue of RDA-eligibility
for extensions to the document defining the extension, with a default being
non-eligible. I hope we can do that without having the versioning RFC and
the NFSv4.x LUB normatively reference one another.

*6.2 RDMA Eligibility*

With regard to your READDIR example, it is true you have no CPU benefit in
this case. The reason RDMA is done is that you only have two transfer
choices in RPC/RDMA, Send and RDMA. If something is too big for your send
buffer, you wind up sending via RDMA, even if that is no more efficient
from a CPU point of view.

*6.3 Binding Specification Completion Assessment*

With regard to the last sentence of the last paragraph, I agree that this
guidance could be helpful but worry that it might be hurtful if done too
rigidly.  In the case of NFSb4 extensions, it is quite natural to present
an extension in the same document as the RDMA-eligible data for that
extension.

---

From: David Noveck <[email-masked]>
Date: Mon, 19 Oct 2015 16:17:29 -0400

> Nothing more. I would
> like to improve the language in bis to make this clear, and
> am open to suggestions.

RFC5666 makes it clear that a chunk does simply represent a string of
XDR-encode bytes, as does your document and your email.

The problem is telling people what it does represent.  RFC5666 and your
document say it represents an argument, a result, an XDR object, etc.

In your email you say something different and it would be helpful to state
this in the document.  As I understand it, an RDMA-eligible item can only
be an opaque array, exclusive of the length.   If that is the case, and I
believe it is, then the cases I was worried about cannot occur.  XDR
objects can be nested , but opaque arrays cannot.

> If there is more than one operation in a compound that has
> a union result with arms that may include an RDMA-eligible
> XDR object, and arms that do not include one, some way will
> need to be found to sort out which write chunk goes with
> which result.

Since interoperable protocol implementations should not pick randomly, I
think the only way to do this is for the first actual RDMA-eligible opaque
array to go with the first write chunk list element, and so on, until you
exhaust one or the other.

On Mon, Oct 19, 2015 at 3:49 PM, Chuck Lever <[email-masked]> wrote:

> I will attempt to address your other comments by revising
> the -02 submitted today. Meanwhile, there are a couple of
> items that need to be discussed further on-list.
>
>
> > On Oct 19, 2015, at 1:50 PM, David Noveck <[email-masked]> wrote:
> >
> > 2.5.3 Read Chunk
> >
> >       • First,, I found the use of the term "argument" confusing.  While
> the data to be written by an NFSv3 WRITE is clearly an argument of the
> WRITE RPC, the situation is different in NFSv4.  The buffer to be written
> by the WRITE operation is not an argument of the COMPOUND RPC.    It is
> part of the argument but is not an argument itself.
>
> This is problematic because RFC 5666 doesn’t make clear that
> the purpose of a chunk is to capture a single opaque XDR
> structure, and not simply an arbitrary group of bytes in the
> XDR stream. This organization makes it straightforward to
> integrate RDMA into existing XDR implementations.
>
> The example of NFSv4 COMPOUND is troubling. The intention of
> both RFC 5666 and 5667 is that a WRITE operation in an NFSv4
> COMPOUND should work just the same way as an NFSv3 WRITE
> operation. The opaque data payload argument, without its
> length, is captured in a read chunk. Nothing more. I would
> like to improve the language in bis to make this clear, and
> am open to suggestions.
>
>
> >       • The possibility that RPC object can be nested within one
> another, means that the same XDR position can be associated with multiple
> RPC objects that start at the same position and end in different
> positions.  This possibility needs to be addressed in some way.
>
> The same XDR Position cannot be associated with multiple XDR
> objects. I’m not sure I understand what you mean by “start at
> the same position but end in different positions”. A list of
> read segments can have the same value in their XDR Position
> field, and that means precisely that they are concatenated,
> in list order, by the receiver, into the sink buffer.
>
> This is discussed in “Read Chunk”, and can be amplified in
> the “Read List” section.
>
>
> >       • In cases in which you have a counted array, the XDR position of
> the counted array is the first byte of the length.  on the other hand, for
> There are two interesting examples of a counted array:
>
> 1. A counted array of bytes. In this case, the length is not
> included in the chunk.
>
> 2. A counted array of complex XDR objects. I believe the array
> length is also not included in the chunk in this case. However
> I would argue that complex XDR objects need to be marshaled,
> and therefore would never be eligible for RDMA (except when
> conveyed via a long message).
>
> RFC 5666 already states that the length of an XDR object is
> not included in the chunk. I assumed this applied to arrays.
>
>
> > 2.5.4 Write Chunk
> >
> > It is not made clear here (or in 2.5.6) how the appropriate
> result/object is selected, given that the write chunk doesn't have an XDR
> position.  I assume that successive list entries are assigned to successive
> stated explicitly.
> Good point, and your assumption that the ULB plays a key role
> here is correct, because there is no Position field in write
> chunks.
>
>
> > Also the ULB gets to decide where in the object the RDMA-eligible piece
> is (i.e. it need to address the write-chunk version of the counted array
> issue).
>
> Again, a counted array is exactly what we have with an NFS
> READ payload, and the array length is transferred inline
> while the payload bytes are transferred in a write chunk.
>
> If you still believe the I-D is inadequate, please do
> suggestion some edits, and I will consider them.
>
>
> > 3.7 Write List Error Cases
> >
> Each NFSv4 COMPOUND is an array of switched unions.
>
> If there is more than one operation in a compound that has
> a union result with arms that may include an RDMA-eligible
> XDR object, and arms that do not include one, some way will
> need to be found to sort out which write chunk goes with
> which result.
>
> Multiple READ_PLUS operations in a compound is the case I’m
> thinking of.
>
> --
> Chuck Lever
> [email-masked]
>
>
>
>

---

From: Chuck Lever <[email-masked]>
Date: Mon, 19 Oct 2015 17:14:46 -0400

> On Oct 19, 2015, at 4:17 PM, David Noveck <[email-masked]> wrote:
> 
> > Nothing more. I would
> RFC5666 makes it clear that a chunk does simply represent a string of XDR-encode bytes, as does your document and your email.
> 
> The problem is telling people what it does represent.  RFC5666 and your document say it represents an argument, a result, an XDR object, etc.
> 
> In your email you say something different and it would be helpful to state this in the document.  As I understand it, an RDMA-eligible item can only be an opaque array, exclusive of the length.   If that is the case, and I believe it is, then the cases I was worried about cannot occur.  XDR objects can be nested , but opaque arrays cannot.

RFC 5666 says specifically that RPC-over-RDMA does not limit
the complexity of what can go into a chunk:
 
> Semantically speaking, the protocol has no restriction regarding data
> types that may or may not be represented by a read or write chunk.
> In practice however, efficiency considerations lead to the conclusion
> that certain data types are not generally "chunkable".  Typically,
> only those opaque and aggregate data types that may attain
> substantial size are considered to be eligible.

and further:
 
> The eligibility of XDR data items to be candidates for being moved as
> data chunks (as opposed to being marshaled inline) is not specified
> by the RPC-over-RDMA protocol.  Chunk eligibility criteria MUST be
> determined by each upper-layer in order to provide for an
> interoperable specification.

What does restrict what can go into a chunk is the ULB.
So, it may be appropriate for RFC 5666bis simply to state
that how a complex data type is chunked must be specified
in a ULB. Do you feel that is enough?

The remaining question is how to make it clear that chunked
XDR objects are atomic (ie, cannot be broken apart or
combined arbitrarily when they are chunked).

As for the length of a complex object:

> When any data within a message is provided via either read or write
> chunks, the chunk itself refers only to the data portion of the XDR
> stream element.  In particular, for counted fields (e.g., a "<>"
> encoding) the byte count that is encoded as part of the field remains
> in the XDR stream, and is also encoded in the chunk list.

“remains in the XDR stream” here means “remains inline.”

So with any counted array, the number of elements goes inline,
and the elements themselves go in the chunk.

—
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 20 Oct 2015 07:41:45 -0400

> combined arbitrarily when they are chunked).

How about:

Each particular XDR aggregate (e.g. union, struct, array, list) may be
designated by the ULB as being represented by a chunk.  Two such objects
may not be combined in the same chunk unless they are the totality of the
contents of a single object that includes them both and nothing else. When
a particular XDR construct is represented by a chunk, all of the words
corresponding to that object are excised from the XDR stream and appear
elsewhere as designated by the chunk lists.  An exception concerns the case
of counted arrays.  If the XDR construct in question* is a counted array *[see
below], the length word is not excised and remains in the XDR stream, while
the array elements themselves appear elsewhere as designated by chunk lists.

It is the responsibility of the ULB, in making such designations, to ensure
that if one XDR construct is embedded within another, that at most one of
these two constructs is to be represented by a chunk.  This ensures that
the mapping from XDR position to the XDR construct represented is
unambiguous.

As to the [see below], consider:

struct abra {

int       ca<*>;

char    dabra<*>;

};

If a struct abra is a chunk, how many length fields appear in the XDR
stream proper?

0 --> "contains a counted array" (also "length word" --> "length word(s)")

1 --> "has a counted array at the start"

2 --> "is a counted array",  as above.

Such cases are unlikely to arise, but the spec still needs to be
unambiguous about this.

On Mon, Oct 19, 2015 at 5:14 PM, Chuck Lever <[email-masked]> wrote:

> > In your email you say something different and it would be helpful to
> state this in the document.  As I understand it, an RDMA-eligible item can
> only be an opaque array, exclusive of the length.   If that is the case,
> and I believe it is, then the cases I was worried about cannot occur.  XDR

---

From: Chuck Lever <[email-masked]>
Date: Tue, 20 Oct 2015 16:57:56 -0400

> 6.3 Binding Specification Completion Assessment
> 
> With regard to the last sentence of the last paragraph, I agree that this guidance could be helpful but worry that it might be hurtful if done too rigidly.  In the case of NFSb4 extensions, it is quite natural to present an extension in the same document as the RDMA-eligible data for that extension. 

I had something else in mind here, and probably expressed
it too vaguely. Since you didn’t recommend any particular
substitution text, I’ll reply on-list.

It’s not a matter of making a hard delineation between
physical documents, but rather making sure we understand
exactly what constitutes a completed ULB specification,
whether it’s part of another document, or stands by
itself (either approach is valid, IMO).

The RDMA-related features of NFSv4.1 are spread between
RFC 5661 and 5667. Would any of the RDMA-related text
in 5661 be considered part of an Upper Layer Binding? If
so, that would be inconsiderate and confusing to
implementers. I’m guessing the authors of both RFCs
got this right, but there’s no guidance to help future
authors.

And the converse problem: we want to avoid adding a few
RDMA gimmes in an extension I-D and calling that a
completed ULB.

—-
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 20 Oct 2015 18:35:57 -0400

> exactly what constitutes a completed ULB specification,

Sounds like a good thing.

I think the next iteration of the versioning document should say
that, for any subsequently published minor version definition
document, there must be a completed ULB, as defined by
the definitive definition of ULB requirements (in rfc5666bis or something
very much like it), and that ULB can be broken up among:

   - A ULB document for a previous minor version
   - Information in feature specification documents
   - Information in the minor version definition document
   - Information in a version-specific ULB document normatively referenced
   by the version specification document.

> in 5661 be considered part of an Upper Layer Binding?

As far as I can determine, no.

> RDMA gimmes in an extension I-D and calling that a

I think that any such would be a behavioral change.  Maybe
there needs to be some additional text in the versioning
document to make this clearer.

The important point here is that an extension cannot make a
behavioral change that applies to existing operations.  That
would require a minor version.   And someone cannot
legitimately call that a ULB unless it meets the requirements for
completed ULB's, and if it is one, it would only apply to the current
minor version and not to previous minor versions.

On Tue, Oct 20, 2015 at 4:57 PM, Chuck Lever <[email-masked]> wrote:

> > With regard to the last sentence of the last paragraph, I agree that
> this guidance could be helpful but worry that it might be hurtful if done
> too rigidly.  In the case of NFSb4 extensions, it is quite natural to
> present an extension in the same document as the RDMA-eligible data for
> that extension.
>

---

From: William Allen Simpson <[email-masked]>
Date: Fri, 30 Oct 2015 12:14:54 -0400

On 10/19/15 1:50 PM, David Noveck wrote:
> I think we need to reach a rough  consensus regarding the future of RPC/RDMA.  Despite the strength of Chuck's arguments regarding the need to do an rfc5666bis, we need to actually come to a decision about that and I think we need to come to decision on
> this fairly soon.
>
I'm an RFC5666 implementer.  Matt Benjamin and I participated
remotely in the prior IETF meeting, where Chuck proposed a list of
options going forward.

It appeared that there was strong support for doing RFC5666bis....

Can the (co-)chair please add it to whatever official list of work
items is maintained, so that we don't perpetually rehash the same
decision process?

> As a way of beginning the process leading to that decision let me state my suggestions about what I think we should decide:
>
>   * As a way of encouraging working group involvement in this area, I suggest that we ask the working group members to review Chuck's document and then ask Chuck to incorporate the resulting input into a draft-ietf-rfc5666-implementation-experience-00.

I also encourage WG members to read it.

Traditionally, going back to my earliest RFC experience in the '80s,
this kind of experience document was a personal RFC, not a
WG-produced RFC.  The current draft title should remain unchanged.
This also facilitates the tools diff and maintaining draft history.

If somebody has an opposing view, they are welcome to do so as
another personal RFC under IETF publication rules.

>   * At some point thereafter, if there is a working group consensus to proceed along that path, we should start work on a draft-ietf-rfc5666bis
>
I believe we have consensus already that RFC5666bis should be a
WG document.  I have volunteered to help Chuck draft it.  There
is only 1 other person whom I know to be an implementer.  At the
recent BaT, Tom Talpey seemed copacetic to implementers revising
the RFC.  We should invite her as a co-author, too.

As we are not meeting next week, there is nothing preventing our
private work on RFC5666bis Internet Standard in preparation for the
next opening of the internet-draft repository.

---

From: Chuck Lever <[email-masked]>
Date: Fri, 30 Oct 2015 12:25:43 -0400

> On Oct 30, 2015, at 12:14 PM, William Allen Simpson <[email-masked]> wrote:
> 

The sentences here were transposed. This should read:

“There is only 1 other person whom I know to be an implementer.
We should invite her as a co-author too.

At the recent BaT, Tom Talpey seemed copacetic to implementers
revising the RFC.”

The implementer mentioned above is currently not a participant
in or contributor to the IETF. I can discuss this with her, but
she may be happy to remain as an acknowledged collaborator.

> next opening of the internet-draft repository.

-—
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 30 Oct 2015 13:44:05 -0400

> > I think we need to reach a rough  consensus regarding the future of
RPC/RDMA.  Despite the strength
> > of Chuck's arguments regarding the need to do an rfc5666bis, we need to
actually come to a decision
> > about that and I think we need to come to decision on
> > this fairly soon.
>
>

> It appeared that there was strong support for doing RFC5666bis....

I agree but I don't know if it amounts to "consensus".  It probably does
but I can't say that there is one with any assurance,

> items is maintained,

I don't think there is such a list, apart from the list of working group
documents actually submitted and being worked on.

I think the practice has been that someone discusses with Spencer the idea
of producing a draft-ietf-nfsv4-something-or-other, and then when it is
submitted,
something-or-other is thereby on the list of documents of being worked on.

> so that we don't perpetually rehash the same
> decision process?

Some day I'm going to write an I-D entitled "Ground Hog Day at the IETF" :-)

> > As a way of beginning the process leading to that decision let me state
my suggestions about what I think we should decide
> >
> > * As a way of encouraging working group involvement in this area, I

> WG-produced RFC.

Although entitled as an experience document, my guess is that
Chuck's document is different from other such documents.  It is a
roadmap for fixing RFC5666.  Also, we've had a number of
WG documents which are not WG-produced in the way that
RFC3530 and draft-ietf-nfsv4-minorversion2 were.

> The current draft title should remain unchanged.

OK with me.  My basic concern is that we maintain momentum on
this by making some RDMA-related document a WG item.  I'm OK if
that turns out to be draft-ietf-nfsv4-rfc5666bis-00.

> WG document.

So I propose that if anyone thinks there is not a consensus in this regard,
he
should send a message to the list so that we can discuss this issue.

On Fri, Oct 30, 2015 at 12:14 PM, William Allen Simpson <
[email-masked]> wrote:

>>   * As a way of encouraging working group involvement in this area, I
>   * At some point thereafter, if there is a working group consensus to

---

From: Chuck Lever <[email-masked]>
Date: Fri, 30 Oct 2015 16:48:12 -0400

> On Oct 30, 2015, at 1:44 PM, David Noveck <[email-masked]> wrote:
> 
> > > I think we need to reach a rough  consensus regarding the future of RPC/RDMA.  Despite the strength 
> > > of Chuck's arguments regarding the need to do an rfc5666bis, we need to actually come to a decision 
> of producing a draft-ietf-nfsv4-something-or-other, and then when it is submitted,
> > > As a way of beginning the process leading to that decision let me state my suggestions about what I think we should decide
> > >
> > > * As a way of encouraging working group involvement in this area, I suggest that we ask the working group members to review Chuck's document 
> 
> > > and then ask Chuck to incorporate the resulting input into a draft-ietf-rfc5666-implementation-experience-00.
> 

I’m OK with either a personal draft or a WG I-D.

As I understand it, a personal draft simply reflects
the opinion of its authors. Identifying requirements
for an update of rfc5666 is something that can remain
as a personal draft.

However, the value of taking this Informational draft
through the WG process is that we might quickly have
a document that current and soon-to-be implementers
can refer to, blessed by WG review, as they wrestle
with RFC 5666 and their own implementations. In
other words, it is no longer the mere opinion of one
author.

Meanwhile the WG can do a thorough rewrite of rfc5666,
the result of which might become a fresh Proposed
Standard.

I don’t think this doc should become rfc5666bis, but
even if it is published, moving forward with that
should not block starting on rfc5666bis.

One purpose of rfc5666bis is to prepare for a full
specification of NFSv4.1 on RDMA. Moving
draft-ietf-rpcrdma-bidirection into WG LC and beyond
_should_ block rfc5666bis IMO because I believe the
material in that draft is a requirement for NFSv4.1
on RDMA and should be included in rfc5666bis.

> So I propose that if anyone thinks there is not a consensus in this regard, he
> On Fri, Oct 30, 2015 at 12:14 PM, William Allen Simpson <[email-masked]> wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

—
Chuck Lever

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 30 Oct 2015 21:37:57 +0000

I find this text in the minutes from the Dallas meeting:

"AI: Tom/Chuck Communicate that we will bis both docs to unsnarl incompatibility"

I assume that this refers to RFC5666 and RFC5667.

Chuck and Tom, can you clarify this for me?

From the above, I will assume (until someone disagrees on-list) that the WG will be building bis documents for both of these RFCs.

To the merits of having Chuck's document move to a WG document....   I don't mind this but I would be working under the assumption that its contents would end up in the above bis documents and would then eventually be "retired" being just a placeholder for the issues and a discussions of the like.

Spencer

-----Original Message-----
From: nfsv4 [mailto:[email-masked]] On Behalf Of Chuck Lever
Sent: Friday, October 30, 2015 1:48 PM
To: David Noveck <[email-masked]>; William Allen Simpson <[email-masked]>
Cc: [email-masked]
Subject: Re: [nfsv4] RPC/RDMA Stuff

> > > *Regarding the Need for rfc5666bis* I think we need to reach a 
> > > rough  consensus regarding the future of RPC/RDMA.  Despite the 
> > > strength of Chuck's arguments regarding the need to do an 
> > > rfc5666bis, we need to actually come to a decision about that and 
> > > I think we need to come to decision on this fairly soon.
> 
> I agree but I don't know if it amounts to "consensus".  It probably 
> does but I can't say that there is one with any assurance,
> 
> I don't think there is such a list, apart from the list of working 
> group documents actually submitted and being worked on.
> 
> I think the practice has been that someone discusses with Spencer the 
> idea of producing a draft-ietf-nfsv4-something-or-other, and then when 
> it is submitted, something-or-other is thereby on the list of documents of being worked on.
> 
> > so that we don't perpetually rehash the same decision process?
> 
> > > As a way of beginning the process leading to that decision let me 
> > > state my suggestions about what I think we should decide
> > >
> > > * As a way of encouraging working group involvement in this area, 
> > > I suggest that we ask the working group members to review Chuck's 
> > > document
> 
> Although entitled as an experience document, my guess is that Chuck's 
> document is different from other such documents.  It is a roadmap for 
> fixing RFC5666.  Also, we've had a number of WG documents which are 
> not WG-produced in the way that
> OK with me.  My basic concern is that we maintain momentum on this by 
> making some RDMA-related document a WG item.  I'm OK if that turns out 
> to be draft-ietf-nfsv4-rfc5666bis-00.

As I understand it, a personal draft simply reflects the opinion of its authors. Identifying requirements for an update of rfc5666 is something that can remain as a personal draft.

However, the value of taking this Informational draft through the WG process is that we might quickly have a document that current and soon-to-be implementers can refer to, blessed by WG review, as they wrestle with RFC 5666 and their own implementations. In other words, it is no longer the mere opinion of one author.

Meanwhile the WG can do a thorough rewrite of rfc5666, the result of which might become a fresh Proposed Standard.

I don’t think this doc should become rfc5666bis, but even if it is published, moving forward with that should not block starting on rfc5666bis.

One purpose of rfc5666bis is to prepare for a full specification of NFSv4.1 on RDMA. Moving draft-ietf-rpcrdma-bidirection into WG LC and beyond _should_ block rfc5666bis IMO because I believe the material in that draft is a requirement for NFSv4.1 on RDMA and should be included in rfc5666bis.

> > I believe we have consensus already that RFC5666bis should be a WG 
> > document.
> 
> So I propose that if anyone thinks there is not a consensus in this 
> regard, he should send a message to the list so that we can discuss this issue.
> 
> RPC/RDMA.  Despite the strength of Chuck's arguments regarding the need to do an rfc5666bis, we need to actually come to a decision about that and I think we need to come to decision on this fairly soon.
> 
> I'm an RFC5666 implementer.  Matt Benjamin and I participated remotely 
> in the prior IETF meeting, where Chuck proposed a list of options 
> going forward.
> 
> this kind of experience document was a personal RFC, not a WG-produced 
> RFC.  The current draft title should remain unchanged.
> If somebody has an opposing view, they are welcome to do so as another 
> personal RFC under IETF publication rules.
> 
> 
> I believe we have consensus already that RFC5666bis should be a WG 
> document.  I have volunteered to help Chuck draft it.  There is only 1 
> other person whom I know to be an implementer.  At the recent BaT, Tom 
> Talpey seemed copacetic to implementers revising the RFC.  We should 
> invite her as a co-author, too.
> 
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.i
> etf.org%2fmailman%2flistinfo%2fnfsv4&data=01%7c01%7csshepler%40microso
> ft.com%7cb79c6f99a891402e916608d2e16b83da%7c72f988bf86f141af91ab2d7cd0
> 11db47%7c1&sdata=xXtTDRzqVA4aF%2bTpInZBeHpInoHWUEvAerCO5uhXzB4%3d

—
Chuck Lever

_______________________________________________
nfsv4 mailing list
[email-masked]
https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fwww.ietf.org%2fmailman%2flistinfo%2fnfsv4%0a&data=01%7c01%7csshepler%40microsoft.com%7cb79c6f99a891402e916608d2e16b83da%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=vWfSotAnJoRXfX7str5FJwPL73t2ql3qPD7oT3h7WcY%3d

---

From: Chuck Lever <[email-masked]>
Date: Fri, 30 Oct 2015 18:17:42 -0400

> On Oct 30, 2015, at 5:37 PM, Spencer Shepler <[email-masked]> wrote:
> 
> 

Speaking for myself: yes, we intended to bis both RFC 5666
and RFC 5667. Tom suggested recently that we could update
only Section 5 of RFC 5667, though I prefer not splitting
the NFS ULB over multiple documents.

I’ve previously proposed a road map for this work:

http://www.ietf.org/mail-archive/web/nfsv4/current/msg13963.html

There were no subsequent objections.

> To the merits of having Chuck's document move to a WG document....   I don't mind this but I would be working under the assumption that its contents would end up in the above bis documents and woul…

Indeed, the text in the current I-D is intended to be
reused liberally, though the organization of that
material would of course be different in a standards
document.

I’m comfortable with a statement of retirement when
there is a Proposed Standard. However . . .

The value of a published Informational document does
wane after a fresh Proposed Standard is available. An
Informational document could even be confusing in the
long run.

In March we chose to pursue a single new rfc5666bis
document so implementers could have a single trusted
document to refer to. A personal draft would expire
automatically as soon as it is no longer renewed,
and disappear conveniently from sight.

—
Chuck Lever

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 30 Oct 2015 22:20:10 +0000

Thanks for the refresher on your earlier email, Chuck.  No reason to wait on the bis work based on that and subsequent response.

Please submit your next I-D as a WG draft and we will keep it as such until the other work is done, etc. as discussed below.

Spencer

-----Original Message-----
From: Chuck Lever [mailto:[email-masked]] 
Sent: Friday, October 30, 2015 3:18 PM
To: Spencer Shepler <[email-masked]>
Cc: David Noveck <[email-masked]>; William Allen Simpson <[email-masked]>; [email-masked]
Subject: Re: [nfsv4] RPC/RDMA Stuff

> Chuck and Tom, can you clarify this for me?

Speaking for myself: yes, we intended to bis both RFC 5666 and RFC 5667. Tom suggested recently that we could update only Section 5 of RFC 5667, though I prefer not splitting the NFS ULB over multiple documents.

https://na01.safelinks.protection.outlook.com/?url=http%3a%2f%2fwww.ietf.org%2fmail-archive%2fweb%2fnfsv4%2fcurrent%2fmsg13963.html&data=01%7c01%7csshepler%40microsoft.com%7ccdf47b96658a4c13de1d08d2e177f09c%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=KHt5Jb5JKGAdMn553CgescIG91S9saAxH1enjxzPNd0%3d

> To the merits of having Chuck's document move to a WG document....   I don't mind this but I would be working under the assumption that its contents would end up in the above bis documents and woul…

Indeed, the text in the current I-D is intended to be reused liberally, though the organization of that material would of course be different in a standards document.

I’m comfortable with a statement of retirement when there is a Proposed Standard. However . . .

The value of a published Informational document does wane after a fresh Proposed Standard is available. An Informational document could even be confusing in the long run.

In March we chose to pursue a single new rfc5666bis document so implementers could have a single trusted document to refer to. A personal draft would expire automatically as soon as it is no longer renewed, and disappear conveniently from sight.

>>>> rfc5666bis, we need to actually come to a decision about that and I 
>> idea of producing a draft-ietf-nfsv4-something-or-other, and then 
>> when it is submitted, something-or-other is thereby on the list of documents of being worked on.
>> 
>> making some RDMA-related document a WG item.  I'm OK if that turns 
>> out to be draft-ietf-nfsv4-rfc5666bis-00.
> 
>> document.  I have volunteered to help Chuck draft it.  There is only 
>> 1 other person whom I know to be an implementer.  At the recent BaT, 
>> Tom Talpey seemed copacetic to implementers revising the RFC.  We 
>> should invite her as a co-author, too.
>> 
>> ft.com%7cb79c6f99a891402e916608d2e16b83da%7c72f988bf86f141af91ab2d7cd
>> 0 11db47%7c1&sdata=xXtTDRzqVA4aF%2bTpInZBeHpInoHWUEvAerCO5uhXzB4%3d
> 
> —
> Chuck Lever
> 
> 
> 
> etf.org%2fmailman%2flistinfo%2fnfsv4%0a&data=01%7c01%7csshepler%40micr
> osoft.com%7cb79c6f99a891402e916608d2e16b83da%7c72f988bf86f141af91ab2d7
> cd011db47%7c1&sdata=vWfSotAnJoRXfX7str5FJwPL73t2ql3qPD7oT3h7WcY%3d

—
Chuck Lever