Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5667bis-06.txt

From: [email-masked]
Date: Fri, 24 Feb 2017 08:50:16 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Network File System Version 4 of the IETF.

        Title           : Network File System (NFS) Upper Layer Binding To RPC-Over-RDMA Version One
        Author          : Charles Lever
	Filename        : draft-ietf-nfsv4-rfc5667bis-06.txt
	Pages           : 19
	Date            : 2017-02-24

Abstract:
   This document specifies Upper Layer Bindings of Network File System
   (NFS) protocol versions to RPC-over-RDMA Version One.  Upper Layer
   Bindings are required in order to enable RPC-based protocols such as
   NFS to use Direct Data Placement on RPC-over-RDMA Version One.  This
   document obsoletes RFC 5667.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5667bis/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5667bis-06

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5667bis-06

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: Chuck Lever <[email-masked]>
Date: Fri, 24 Feb 2017 09:00:27 -0800

> Begin forwarded message:
> 
> From: [email-masked]
> Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5667bis-06.txt
> Date: February 24, 2017 at 8:50:16 AM PST
> To: <[email-masked]>
> Cc: [email-masked]
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

Changes since -05:

Dave Noveck observed that my stated goal was "some
degree of transport version agnosticism" but there
is still inconsistent language throughout the document.
I've been struggling with how to discuss certain
mechanisms, such as short Reply chunk retry, without
the mention of version-specific transport protocol
elements.

Thus I've decided to reverse course and explicitly
state this document applies to RPC-over-RDMA Version
One, though not explicitly exclude the use of the ULB
by future versions or extensions.

  o A ULB sometimes has to discuss in detail how a ULP
    might use a transport. For example, it might need
    to refer to a specific error code or transport
    mechanism that is available only in some particular
    transport versions. That's challenging to do when
    using only abstract terms.

  o It is not possible to predict how future transports
    might innovate. Specificity in the text of a ULB can
    make it difficult to apply easily to later versions
    of a transport, or its extensions.
       + For instance, there are a couple places (like
         dealing with READ_PLUS replies, or how to handle
         multiple Write chunks) where there is a very
         transport-specific mechanism described. We could
         have some innovation in the future to help deal
         with these situations.

   o The change in direction resolves some subtle issues
     about how cross-citation will work when defining a
     transport version or extension and possibly a ULB to
     go with it. It appears more straightforward to me to
     provide a basic ULB that future transport and
     extensions can point to and say either:
       + That ULB rule can be re-used, or
       + That ULB rule can be re-used with these changes
         or exceptions, or
       + This is the new ULB rule that must be used for
         this new mechanism

This might be a controversial change, but I feel the
document is more clear now while still enabling its use
for future transports and their extensions.

Karen Deitke had questions about whether chunk reduction
can be used with Long messages. The recent removal of "MUST
ignore subsequent chunks" seems to have opened a little
gap. I've added some language which I hope closes that gap.

While working with IANA on rfc5666bis, Tom Talpey noticed
that the official IANA registration of the nfsrdma ports
references RFC 5666, but should reference the NFS ULB. RFC
5667 doesn't have an IANA Considerations section, and the
mention of nfsrdma in RFC 5666 is as an example. Thus the
IANA registry citation needs to be corrected, and one of
these documents needs to have an official specification
of nfsrdma. To make the IANA registry and the updated
documents consistent, I've updated the IANA Considerations
sections in both rfc5666bis and rfc5667bis.

Karen and Dave both suggested a reorganization of the
discussion of short Reply chunk retry.

A very recent discussion on [email-masked]
brought up the question of whether the language of RFC
5661 Section 2.9.1 forecloses on the use of RPC-over-RDMA
with NFS version 4, because of version 4's tighter
requirements for transport-level congestion avoidance.
Of special concern was RoCEv2, which is implemented over
UDP on IP. A section has been introduced to discuss this,
and summarize the Working Group's (hopefully) approval of
the use of RPC-over-RDMA for NFS version 4 deployments.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 24 Feb 2017 17:30:32 -0500

Let's start with the most controversial area and try to put that aside for
the moment.  Chuck and I continue to have different opinions about the
appropriate way for ULBs (all of which that exist are in the current
document) to deal with the fact that, NFS-over-RDMA is no longer solely
represented by Version One.  Chuck and I have had extensive discussions and
decided that we could not resolve this in a reasonable time.   As a result,
I accepted Chuck's decision to proceed with rfc5667bis on a
Version-One-only basis and agreed to review -06 on that basis.

Others in the working group are free to take a different view but I feel
that it would not be helpful to delay rfc5667bis because of this issue.
The working group reached consensus on the original versions these
documents about a decade ago and only now are we reaching an equivalent
state with the revived documents.  Further transport versions are needed to
get us to the protocol we want, but the working group has put a priority on
reviving Version One and I would urge people not to add unnecessarily to
the delay in getting that part of our work done.

Later, I will be submitting a draft-dnoveck-nfsv4-nfsulb that illustrates
how a more transport-generic NFS ULB might be written.  Chuck and I have
repeatedly disagreed about how difficult (or even possible) it was to do
this.  It is hard to discuss this sort of issue in the abstract.  However,
once we have a document to look at and review, the working group can see
for itself whether this is a viable approach and decide if it might be more
appropriate in the long term, as the transport is extended.  This document
is based on rfc5667bis so that Chuck would normally be listed as a
co-author, but Chuck has decided against that.  I hope he will reconsider.

---

From: David Noveck <[email-masked]>
Date: Fri, 24 Feb 2017 17:32:21 -0500

Others in the working group are free to take a different view but I feel
that it would not be helpful to delay rfc5666bis because of this issue.
The working group reached consensus on these documents about a decade ago
and only now are we reaching an equivalent state with the revived
documents.  Further transport versions are needed to get us to the protocol
we want, but the working group has put a priority on reviving Version One
and I would urge people not to add unnecessarily to the delay in getting
that part of our work done.

I will be submitting a draft-dnoveck-nfsv4-nfsulb that illustrates how a
more transport-generic NFS ULB might be written.  Chuck and I have
repeatedly disagreed about how difficult (or even possible) it was to do
this.  It is hard to discuss this issue in the abstract so that once we
have a document to look at and review, the working group can see for itself
whether this is a viable approach and decide if it might be more
appropriate in the long term, as the transport is extended.  This document
is based on rfc5667bis so that Chuck would normally be listed as a
co-author, but Chuck has decided against that.  I hope he will reconsider.

---

From: Chuck Lever <[email-masked]>
Date: Sat, 25 Feb 2017 08:22:52 -0800

> On Feb 24, 2017, at 2:32 PM, David Noveck <[email-masked]> wrote:
> 
> Let's start with the most controversial area and try to put that aside for the moment.  Chuck and I continue to have different opinions about the appropriate way for ULBs (all of which that exist are in the current document) to deal with the fact that, NFS-over-RDMA is no longer solely represented by Version One.  Chuck and I have had extensive discussions and decided that we could not resolve this in a reasonable time.   As a result, I accepted Chuck's decision to proceed with rfc5667bis on a Version-One-only basis and agreed to review -06 on that basis.
> 
> Others in the working group are free to take a different view but I feel that it would not be helpful to delay rfc5666bis because of this issue.   The working group reached consensus on these documents about a decade ago and only now are we reaching an equivalent state with the revived documents.  Further transport versions are needed to get us to the protocol we want, but the working group has put a priority on reviving Version One and I would urge people not to add unnecessarily to the delay in getting that part of our work done.
> 
> I will be submitting a draft-dnoveck-nfsv4-nfsulb that illustrates how a more transport-generic NFS ULB might be written.  Chuck and I have repeatedly disagreed about how difficult (or even possible) it was to do this.  It is hard to discuss this issue in the abstract so that once we have a document to look at and review, the working group can see for itself whether this is a viable approach and decide if it might be more appropriate in the long term, as the transport is extended.

The function of a ULB is to adapt the use of specific
transport mechanisms to an Upper Layer Protocol. It's a
practical document written for implementers.

There is more to a ULB than noting which XDR data items
are DDP-eligible.

The current NFS ULB has several sections that account for
deficiencies in RPC-over-RDMA Version One. If the WG designs
subsequent transport protocols correctly, future ULBs will
likely not need to address some of these deficiencies, or
could need to address them in a different way.

Future ULBs might need new discussion to address mechanisms
that we can't now predict. We have right now only one extant
protocol version, one extant ULB, and one direct data
placement mechanism. There are some personal drafts that
discuss possible futures, but IMO the WG can't build a
Standards Track document on those ideas until they
themselves are advanced to RFC documents.

So I ask: Based on the flawed ULP and transport we have
today and this set of speculative proposals, how will the
WG provide a guarantee of a lasting and high quality
agnostic ULB?

There are a long list of tasks ahead of us, including
implementing prototypes and productizing new protocols. Is
an agnostic ULB going to benefit implementers directly, or
are implementers not the target audience of such a document?

I'd rather keep extension and ULB documents small and narrow
in focus. That way they are easier to author, review, and
implement, and the ULBs can discuss transport and ULP issues
with appropriately precise language about specific well-
documented protocols.

If the WG creates an agnostic ULB, it makes sense as an
informational document that feeds the process of protocol
design. In that case, let's not call it a ULB at all.

> This document is based on rfc5667bis so that Chuck would normally be listed as a co-author, but Chuck has decided against that.  I hope he will reconsider.
> 
> 
> 
> On Fri, Feb 24, 2017 at 12:00 PM, Chuck Lever <[email-masked]> wrote:
> 
> 
> > Begin forwarded message:
> >

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sat, 25 Feb 2017 16:44:04 -0500

> practical document written for implementers.

It is also a standards-track document that is, according to
rfc5666bis, required for interoperability, and for good reason.

> There is more to a ULB than noting which XDR data items

Yes, but things have varied over time.  RFC5667 had tons
of material that duplicated what was in RFC5666.  You got
rid of that stuff in rfc5667bis and the document is the better
for it.

The only things that have to be in a ULB are the specific things
noted as required in rfc5666bis, if it is approved.

It might possible to split the material into a standards-track
ULB and an informational document about implementation
issues with regard to the interaction of specific protocols with
specific transport versions or extensions.  However, I don't
see that happening.  For the foreseeable future, I think ULBs
will continue to contain the standards-track elements required
by rfc5666bis, as well as additional explanatory material in a
supporting role.

> deficiencies in RPC-over-RDMA Version One.

That's true.  A good part of those deal with the unfortunate consequences
of designing a protocol extension providing for the transfer of bulk data
i.e. READ_PLUS, without any attention at all to the implications for
RPC-over-RDMA.  You then had to clean up that mess.

> If the WG designs

If by "correctly" you mean "as if we had a clean sheet of paper"
then this is so, but we need to proceed on the assumption that
a clean sheet of paper is unlikely to be available.  As a result,
although one could address READ_PLUS in a different way,
assuming the facilities in rtrext were always available,
draft-dnoveck-nfsv4-nfsulb assumes that these restrictions will
be kept indefnitely, as there doesn't seem to be any good reason
in incur more complexity to change this.  The reasoning is explained
in nfsulb.

> that we can't now predict.

Since one can't know what one can't predict, it is best to proceed on the
basis of what one might reasonably be expected to happen.

> We have right now only one extant
> placement mechanism.

Actually we have one ULB document which contains within it
five ULBs but who's counting :-)

> There are some personal drafts that

As stated, this seems to be a recipe for inaction, although
the problem would be worse in your ULB model than in the case
of a generic NFS ULB.

Suppose one develops an extension X (or transport
version N) . During the discussion, development and
review of such a transport feature/extension,  a necessary
concern will be on how this affects NFS.  So one cannot
advance the document without the assurance that any NFS
bases are covered.

So, if you need to address NFS issues to make progress,
and can't write the ULB as a working group draft until the
transport part becomes an RFC, we are essentially stuck.
Note that the gap from WGLC to RFC Publication is something
like a year.

It makes more sense to treat these issues in parallel, making sure that
there is appropriate ULB support while advancing the transport work.
In some cases the transport improvement would become a working
group document at the same time as the needed ULB support since
it is hard to consider the transport issues without knowing how they
will be addressed in the ULB.  The same applies when going foward
to the IETF.

> WG provide a guarantee of a lasting and high quality

First of all, neither the working group nor the IETF provides
any guarantees.  Everything is done an as-is basis.

One can only do the best one can given what one knows and
reasonably expects.  draft-dnoveck-nfsv4-nfsulb will have
a section explaining why the structure chosen in that document
can reasonably be expected to be long-lasting.

> There are a long list of tasks ahead of us, including/

The target audience for draft-ietf-nfsv4-nfsulb is expected to be
the working group.

If the working group decides to proceed on a transport-generic,
I expect it to benefit the developers by simplifying the issues of
getting a set of documents to proceed with.  Instead of needing a
transport and a corresponding ULB document/addendum, the ULB for
each protocol is fixed and my belief is that that simpler situation will
be beneficial to implementers.

> with appropriately precise language about specific well-

I also prefer smaller documents but as we found out
with the v4.2 experience, shorter documents are not a
magic elixir.

I decided that rtrext would work better as a single document
containing two extensions because so much of the necessary
XDR was shared between the two.  This could be changed
based on working group feedback.

As far as ULB length, nfsulb looks like it will be 25  pages, as
opposed to rfc5667bis-06 which is 19.  That is longer but that
sort of length difference doesn't worry me.

> design.

draft-dnoveck-nfsv4-nfsulb is intended to contribute to the
process of protocol design and can be an informational document.

The intention is to contribute to the development of an
eventual standards-track document to serve as a
transport-agnostic NFS ULB.

> In that case, let's not call it a ULB at all.

Let's not get into a trademark dispute but, if a document
meets the requirements laid out in rfc5666bis, it has to be
called a ULB.

I feel that the working group should do a Generic/agnostic
ULB and believe that it should meet the requirements laid out
rfc5666bis.  If the working group decides to do that, I think the
result pretty much has to be called a ULB since it will be one.

On Sat, Feb 25, 2017 at 11:22 AM, Chuck Lever <[email-masked]>
wrote:

> > Let's start with the most controversial area and try to put that aside
> for the moment.  Chuck and I continue to have different opinions about the
> appropriate in the long term, as the transport is extended.
>
> > This document is based on rfc5667bis so that Chuck would normally be
> listed as a co-author, but Chuck has decided against that.  I hope he will
> reconsider.
> >
> >
> >

---

From: Chuck Lever <[email-masked]>
Date: Sun, 26 Feb 2017 11:37:34 -0800

> On Feb 25, 2017, at 1:44 PM, David Noveck <[email-masked]> wrote:
> 

> by rfc5666bis, as well as additional explanatory material in a

It is precisely that latter part that cannot be
included in a "for all time" NFS ULB. That explanatory
material will likely be different for each transport
protocol.

Issues about reply size estimation, for instance, are
going to be less relevant for a transport that can use
message continuation when the client underestimates the
size of a Reply chunk. But they happen to be relevant
for RPC-over-RDMA Version One.

Do you believe the set of requirements a ULB has to
contain will remain the same forever?

> basis of what one might reasonably be expected to happen.  

No. It's best to leave issues open if we can't know
in advance what innovations might occur.

My biggest worry with a generic ULB is that we would
lock in a bunch of assumptions about how direct data
placement works (mechanism), and thus quell the
possibility of true innovation in the transport layer.

> be beneficial to implementers.

Let's look at this from another standpoint. What you
seem to want to do is have a way of describing which
data items should be directly placed _using_ _any_
_transport_.

Say, TCP, or SCTP. Wouldn't it be nice to allow direct
placement with good ole TCP? Technically, it can be
done, but there's no "ULB" to define which data items
a client or server implementation should prepare for
being placed directly.

Maybe you additionally want to create a way of
decorating XDR definitions to specify which data
items may be placed directly?

And you seem to want to include reply size estimation,
which is something that is potentially important for
building efficient implementations on all transports,
including TCP. For TCP it is not an interop issue,
though it can be for some transports.

Therefore your focus seems to be on how to get better
NFS protocol design and implementation. It's really
not about specific transport mechanisms.

RDMA ULBs are necessarily specific to the underlying
transport for which they are written. When new
transport mechanisms arise, a new ULB document is key
to making use of them.

Please don't take that tool away.

> meets the requirements laid out in rfc5666bis, it has to be 

If you are cleaving to the requirements in rfc5666bis,
it would necessarily be a ULB for RPC-over-RDMA Version
One. If you are going for anything else, then it's not
defined.

So let's take this out of the realm of RPC-over-RDMA
and "Upper Layer Bindings". This is strictly about
the design of NFS: it's about defining policy rather
than mechanism.

> result pretty much has to be called a ULB since it will be one.

My counter-proposal is the WG should sponsor one or
more documents that discuss how to improve the NFS
protocol in the areas of how it would use direct
data placement, and where it can help in the
preparation of transport receive resources.

This is not an Upper Layer Binding because it would
not depend on any particular transport implementation
or mechanism (except, perhaps, for examples), and
because it goes directly to how NFS protocol
elements are conceived and specified.

New ULBs for transports that provide features like
direct data placement could then refer to these
documents.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sun, 26 Feb 2017 20:30:00 -0500

As the working group may have gathered by now, this is a matter on which
Chuck and I have agreed to disagree.  FWIW, we both seem to be keeping up
our ends of the bargain :-)

As I said before, I don't think it is all that productive to argue about
this in a vacuum.   After draft-dnoveck-nfsv4-nfsulb is published (should
be around a week), there will be a better opportunity to continue this
discussion.  Another helpful factor would be to proceed on making new
versions or extensions into working group documents.  At that point we will
have something concrete to discuss as we consider whether versions and
extensions are better served by a model in which many ULBs per protocol
exist or with a more generic approach.  As we now seem to be finishing up
on the working group's work in reviving Version One, it is appropriate to
move on to other work in the RDMA area, although getting rfc5667bis through
WGLC should have priority.

In any case, even if a generic approach is adopted, there shouldn't be
anything that would essentially force the working group to keep it when it
is no longer helpful.  If Chuck, or someone else, can show that a better
job could be done for transport version  with a version-specific model,
there should be nothing stopping the working group from producing that
document and having it update the generic NFS ULB document.  I don't think
that is going to happen but everybody in the working group, including me,
should listen seriously, if that case is made.