Subject: [nfsv4] New draft for working group charter

From: David Noveck <[email-masked]>
Date: Thu, 11 May 2017 07:56:52 -0400

The following is the second iteration of my attempt to provide a draft
charter to be "whacked" (as Beepy put it) on the working group list so all
comments are appreciated.  However, if someone would like to present an
alternate draft, that's OK too.  We do need some sort of draft to work from
for a discussion in Prague.  Without it, a discussion in Prague is not
going to arrive at a useful charter candidate.
This has the following changes from the initial draft.  If you object to
any of these changes, let me know right away:

   - Deletion of References to FedFS, per Chuck's comments.
   - Inclusion of other ONC components in the *Maintenance *section, per
   Chuck's comments.
   - Deletion of the out-of-date sections* RFC5664bis* and *NFSv4.2.*
   - Deletion (for now) of the section *NFSv4 Multi-Domain Access for
   FedFS.*  Unlike the previously mentioned sections, this could come back,
   probably in the form *NFSv4 Multi-Domain Access* if someone (e.g. Andy)
   provides a draft charter section.
   - Added a reference within the *Maintenance *section, to the ability of
   technical updates to NFSv4 versions to include limited XDR changes.
   - A start at a draft* Milestones* section (see below).  This is very
   limited since most WG documents are past WGLC right now.  I've made some
   reasonable assumptions regarding migration-issues.  When -13 is published,
   the way forward in this area will clearer and the list could be updated to
   include some expected documents to address the issues described in the
   Informational document.  BTW, I'm assuming there will be a need for WGLC
   for that document as a means of establishing consensus on a way forward for
   those issues even though I believe there is no point in publishing that
   document as an RFC.
   - In order to help deal with our  limited set of current milestones,
   I've followed Spencer's suggestion and adapted the material he cited from
   the TCPM charter.

Draft Charter for Working Group (Iteration Two)

NFS Version 4 is the IETF standard for file sharing. To maintain NFS
Version 4's utility and currency, the working group is chartered to
maintain the existing NFSv4.0, NFSv4.1, NFSv4.2, protocols and related
specifications of ONC components such as those defining RPC, XDR, and
RPCSECGSS. In addition, extensions will be developed, as necessary, to
correct problems with the protocols as currently specified, to accommodate
needed file system semantics, and to make significant performance
improvements.

*Maintenance*

The working group has found that as NFSv4 implementations mature and
deployments continue, clarifications to existing RFCs are needed. These
clarifications assist vendors in delivering quality and interoperable
implementations. The working group is chartered with the vetting of the
issues and determining correctness of submitted errata. In addition, some
areas may need more concentrated work to correct the specifications
already published or to deal with unanticipated interactions between
features. In the cases in which the required changes are inappropriate for the
errata system, the working group will assist in publication of best
practices RFCs or of RFCs that provide editorial modification or technical
updates to original RFCs.  Once, the new NFSv4 versioning framework is
approved, such technical updates to NFSv4 versions could include limited
XDR changes.

*Extension*

The NFSv4 protocol is designed to allow extension by the addition of new
operations or new attributes, the creation of minor versions, and the
definition of new pNFS mapping types.  The working group will discuss
proposals for such extensions and assure they have adequate technical
review including discussion of their interaction with existing features
before adopting them as working group items and helping to draft
specification documents. Siniilarly, associated ONC protocol components
that have a versioning/extension  framework can be suitably extended to
accommodate new security requirements, and to make significant performance
improvements.

*Performance Challenges*

The increase of network bandwidths and the reduction of latencies
associated with network traffic and access to persistent storage have
created challenges for remote file access protocols which need to meet
increasingly demanding performance expectations.  Some work already done in
this area includes the respecification of RPC-over-RDMA Version One and the
pNFS SCSI layout.  It is lexpected that further work in this area will be
required.  This might take the form of further RPC-over-RDMA versions,
adaptation of the SCSI layout to NVMe, or the development of an
RDMA-oriented pNFS layout type.  The working group needs to discuss these
alternatives, and possibly others, and develop the most promising ones.

Milestones (Preliminary draft)

Because the previous charter was at variance with the work the group was
actually doing, the list of pending milestones that can determined now is
quite limited.  To accommodate this situation and in light of the fact that
maintenance activities are inherently unpredictable, new milestones that
fall within the scope specified within the charter can be added after
working group consensus on acceptance and approval by the responsible Area
Director.

*Date               Milestone*

Nov. 2017      Publication of RFC5666bis as a Proposed Standard

Nov. 2017      Publication of a Proposed Standard describing bidirectional
operation for RPC-over-RDMA

Jan. 2018       Publication of RFC5667bis as a Proposed Standard.

Feb. 2018       Publication of a Proposed Standard describing NFSv4
versioning/extension framework.

Feb. 2018       Publication of a Proposed Standard describing the umask
extension to NFSv4.2.

Feb. 2018       Publication of a Proposed Standard describing the xattr
extension to NFSv4.2.

March 2018    WGLC for draft-ietf-nfsv4-migration-issues (Informational)

---

From: Chuck Lever <[email-masked]>
Date: Thu, 11 May 2017 12:58:04 -0400

> On May 11, 2017, at 7:56 AM, David Noveck <[email-masked]> wrote:
> 
> The following is the second iteration of my attempt to provide a draft charter to be "whacked" (as Beepy put it) on the working group list so all comments are appreciated.  However, if someone would like to present an alternate draft, that's OK too.  We do need some sort of draft to work from for a discussion in Prague.  Without it, a discussion in Prague is not going to arrive at a useful charter candidate.
> 
> This has the following changes from the initial draft.  If you object to any of these changes, let me know right away:
> 	• Deletion of References to FedFS, per Chuck's comments.
> 	• Inclusion of other ONC components in the Maintenance section, per Chuck's comments.
> 	• Deletion of the out-of-date sections RFC5664bis and NFSv4.2.
> 	• Deletion (for now) of the section NFSv4 Multi-Domain Access for FedFS.  Unlike the previously mentioned sections, this could come back, probably in the form NFSv4 Multi-Domain Access if someone (e.g. Andy) provides a draft charter section.
> 	• Added a reference within the Maintenance section, to the ability of technical updates to NFSv4 versions to include limited XDR changes.
> 	• A start at a draft Milestones section (see below).  This is very limited since most WG documents are past WGLC right now.  I've made some reasonable assumptions regarding migration-issues.  When -13 is published, the way forward in this area will clearer and the list could be updated to include some expected documents to address the issues described in the Informational document.  BTW, I'm assuming there will be a need for WGLC for that document as a means of establishing consensus on a way forward for those issues even though I believe there is no point in publishing that document as an RFC.
> 	• In order to help deal with our  limited set of current milestones, I've followed Spencer's suggestion and adapted the material he cited from the TCPM charter.
> NFS Version 4 is the IETF standard for file sharing. To maintain NFS Version 4's utility and currency, the working group is chartered to maintain the existing NFSv4.0, NFSv4.1, NFSv4.2, protocols and related specifications of ONC components such as those defining RPC, XDR, and RPCSECGSS. In addition, extensions will be developed, as necessary, to correct problems with the protocols as currently specified, to accommodate needed file system semantics, and to make significant performance improvements.  
> 
> Maintenance
> 
> The working group has found that as NFSv4 implementations mature and deployments continue, clarifications to existing RFCs are needed. These clarifications assist vendors in delivering quality and interoperable implementations. The working group is chartered with the vetting of the issues and determining correctness of submitted errata. In addition, some areas may need more concentrated work to correct the specifications already published or to deal with unanticipated interactions between features. In the cases in which the required changes are inappropriate for the errata system, the working group will assist in publication of best practices RFCs or of RFCs that provide editorial modification or technical updates to original RFCs.  Once, the new NFSv4 versioning framework is approved, such technical updates to NFSv4 versions could include limited XDR changes.
> 
> Extension
> 
> The NFSv4 protocol is designed to allow extension by the addition of new operations or new attributes, the creation of minor versions, and the definition of new pNFS mapping types.  The working group will discuss proposals for such extensions and assure they have adequate technical review including discussion of their interaction with existing features before adopting them as working group items and helping to draft specification documents. Siniilarly, associated ONC protocol components that have a versioning/extension  framework can be suitably extended to accommodate new security requirements, and to make significant performance improvements.  
> 
> Performance Challenges

Opportunities? Just a thought.

> The increase of network bandwidths and the reduction of latencies associated with network traffic and access to persistent storage have created challenges for remote file access protocols which need to meet increasingly demanding performance expectations.  Some work already done in this area includes the respecification of RPC-over-RDMA Version One and the pNFS SCSI layout.  It is lexpected that further work in this area will be required.  This might take the form of further RPC-over-RDMA versions, adaptation of the SCSI layout to NVMe, or the development of an RDMA-oriented pNFS layout type.  The working group needs to discuss these alternatives, and possibly others, and develop the most promising ones.

Specific milestones should include:

 - Trunking (multi-pathing)
 - NVMe layout type
 - RDMA layout type
 - flexfile layout type

These are either far along, or there are identified authors and energy
and the work is well-scoped. Other work items are less mature.

I don't think the focus of this section should be strictly on RDMA.
We've introduced performance features in other areas of the stack besides
pNFS and RDMA. For example, READ_PLUS, S2SC, and ALLOCATE are intended to
help performance. One of the performance issues we chronically face is
that NFS itself is still chatty (eg. Tom's suggestion about reducing the
round-trip overhead of OPEN with delegation).

There is no mention of making NFS into a premier storage technology for
supporting virtualization. RDMA helps there, as do the new features in
NFSv4.2. New security models are needed to properly handle multi-tenancy,
for example. But we've moved ahead in this area for years (NFSv4.2)
without mention in the WG charter.

> Because the previous charter was at variance with the work the group was actually doing, the list of pending milestones that can determined now is quite limited.  To accommodate this situation and in light of the fact that maintenance activities are inherently unpredictable, new milestones that fall within the scope specified within the charter can be added after working group consensus on acceptance and approval by the responsible Area Director.
> 
> Date               Milestone
> 
> Nov. 2017      Publication of a Proposed Standard describing bidirectional operation for RPC-over-RDMA
> 
> Feb. 2018       Publication of a Proposed Standard describing NFSv4 versioning/extension framework.
> 
> Feb. 2018       Publication of a Proposed Standard describing the umask extension to NFSv4.2.
> 
> Feb. 2018       Publication of a Proposed Standard describing the xattr extension to NFSv4.2. 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 11 May 2017 14:56:47 -0400

> Specific milestones should include:

I want to explain that my list could not include a lot of things you
mention because I felt it best to limit it to WG items.  If there are
documents that become working group items between now and when the charter
is submitted, then they should included.  I'm not sure we would be able to
include things that aren't working group items at charter submission,
although i believe that we would be able to take advantage of the provision
that Spencer S. suggested to allow later addition of milestones :-)

This has no current WG item.   I expect there would be some discussion of
 possible paths forward in this area once migration-issues-13 is
published.  I'm going to be talking about this area at IETF99 and hope that
Andy will be able to as well.   I'm hoping that soon after IETF99 we will
able to add one or more milestones here.

 - NVMe layout type
 - RDMA layout type

For these there is no current WG document or I-d.  I am looking forward to
work being done on these but don't expect these to be included in the
charter submission.

 - flexfile layout type

I did ask Tom privately about setting a target date for this since it is a
WG item.  I anticipate correcting this at IETF99 by guessing at dates until
Tom nods his head.

> These are either far along, or there are identified authors and energy and
the work is well-scoped.

True but I used having a WG document started or at least agreed upon as the
gating criterion.  I'm willing to change that if you suggest an alternative
criterion.

> Other work items are less mature.

I don't agree. Specifically, I feel the following I-d's should be
considered for WG status and that could well happen  before, at, or soon
after IETF-99:

   - draft-cel-nfsv4-rpcrdma-version-two
   - draft-cel-nfsv4-rpcrdma-cm-pvt-msg

If that happens, we could add one or more milestones for those.

> I don't think the focus of this section should be strictly on RDMA.

It isn't.

pNFS and RDMA.

True.

> For example, READ_PLUS, S2SC, and ALLOCATE are intended to
t> hat NFS itself is still chatty (eg. Tom's suggestion about reducing the
> round-trip overhead of OPEN with delegation).

I felt that these changes were dealt with adequately in the Extension
section.  I guess I
could rename "Performance Challenges" to be "Major Performance Challenges".
If this section were to include any performance-related improvements, I
feel it
would lose necessary focus

> supporting virtualization.

No there isn't.  Is there some addition to the charter draft you would like
to suggest?

I certainly approve of the goal.  I just don't think it is charter-ready
yet, but will
look carefully at any suggested text you provide.  We need to explain the
need/justification for this clearly to Beepy and the Spencers.

> RDMA helps there, as do the new features in NFSv4.2.

> New security models are needed to properly handle multi-tenancy,

NFSv4.2 is mentioned in the existing charter.  It is not in my charter
draft,
because that work is done, except for post-RFC7862 extensions to v4.2.

On Thu, May 11, 2017 at 12:58 PM, Chuck Lever <[email-masked]>
wrote:

> >       • Deletion (for now) of the section NFSv4 Multi-Domain Access for
> FedFS.  Unlike the previously mentioned sections, this could come back,
> probably in the form NFSv4 Multi-Domain Access if someone (e.g. Andy)
> >       • Added a reference within the Maintenance section, to the ability
> of technical updates to NFSv4 versions to include limited XDR changes.
> >       • A start at a draft Milestones section (see below).  This is very
> >       • In order to help deal with our  limited set of current
> milestones, I've followed Spencer's suggestion and adapted the material he
> cited from the TCPM charter.
> areas may need more concentrated work to correct the specifications already
> published or to deal with unanticipated interactions between features. In
> the cases in which the required changes are inappropriate for the errata
> system, the working group will assist in publication of best practices RFCs
> or of RFCs that provide editorial modification or technical updates to
> original RFCs.  Once, the new NFSv4 versioning framework is approved, such
> technical updates to NFSv4 versions could include limited XDR changes.
> >
> > Extension
> >
> > Nov. 2017      Publication of a Proposed Standard describing
> bidirectional operation for RPC-over-RDMA
> >

---

From: Chuck Lever <[email-masked]>
Date: Thu, 11 May 2017 16:53:28 -0400

> On May 11, 2017, at 2:56 PM, David Noveck <[email-masked]> wrote:
> 
> I want to explain that my list could not include a lot of things you mention because I felt it best to limit it to WG items.  If there are documents that become working group items between now and when the charter is submitted, then they should included.

Fair enough. We need some gauge of WG consensus, focus,
and energy.

> I'm not sure we would be able to include things that aren't working group items at charter submission, although i believe that we would be able to take advantage of the provision that Spencer S. suggested to allow later addition of milestones :-)
> 
> This has no current WG item.   I expect there would be some discussion of  possible paths forward in this area once migration-issues-13 is published.  I'm going to be talking about this area at IETF99 and hope that Andy will be able to as well.   I'm hoping that soon after IETF99 we will able to add one or more milestones here.
>   
>  - NVMe layout type
>  - RDMA layout type
> 
> For these there is no current WG document or I-d.  I am looking forward to work being done on these but don't expect these to be included in the charter submission.
> 
>  - flexfile layout type
> 
> I did ask Tom privately about setting a target date for this since it is a WG item.  I anticipate correcting this at IETF99 by guessing at dates until Tom nods his head.
> 
> > These are either far along, or there are identified authors and energy and the work is well-scoped.
> 
> True but I used having a WG document started or at least agreed upon as the gating criterion.  I'm willing to change that if you suggest an alternative criterion.
> 
> I don't agree. Specifically, I feel the following I-d's should be considered for WG status and that could well happen  before, at, or soon after IETF-99:
> 	• draft-cel-nfsv4-rpcrdma-version-two
> 	• draft-cel-nfsv4-rpcrdma-cm-pvt-msg

cm-pvt-msg could move forward, but there has been a palpable
objection to making this work an official standard. Thus I
don't feel it is ready to promote without further discussion.

RPC-over-RDMA Version Two could become a large project.

There are some issues that the WG needs to consider
carefully before enabling such an effort, most especially
what is the appetite by vendors/implementers for a new
version of the protocol, but also, who has the resources
to construct prototypes? Do we understand and agree on
what problems we are trying to address in the longer term?
Have we reached the end of RoRV1's ability to deliver
performance?

There have been questions about the scope of RoRV2 for a
while. The answer shifts depending on whether we think
there is or is not a suitable out-of-band mechanism for
enabling Remote Invalidation and large inline thresholds.
To promote this document I would like to have clarification
of the scope of RoRV2; and that in turn requires clarity on
the disposition of cm-pvt-msg.

> I felt that these changes were dealt with adequately in the Extension section.  I guess I
> If this section were to include any performance-related improvements, I feel it 
> No there isn't.  Is there some addition to the charter draft you would like to suggest?

I'm struggling with an existential question.

I'm wondering why there is now a special charter focus on
performance, but not on virtualization, which has been a
de facto focus of WG effort for the past five years.
Why isn't it enough to say "The WG is responsible for
extending these protocols as needed."

If there is a constraint on what may be included in our
charter, it seems like there will be a problem including
performance in particular, as there are no WG documents
specifically about performance.

If protocol innovation is indeed driven by the individuals
on the WG, then it seems to me that "performance" should
not be part of our charter. Improving performance should
be facilitated by the WG / IETF partnership, but driven by
the diaspora.

I'm comfortable with the Maintenance and Extension sections,
and might even add -- for completeness -- that the WG is
responsible for approving changes to certain sections of
the IANA registries.

I am in philosophical agreement with the technical goal of
better performance, and will continue working on it. Still
not sure whether that is something that needs to appear in
the WG charter.

Spencer (D or S) would you happen to have a preferred BCP
or RFC that discusses the requirements of a WG charter?

> I certainly approve of the goal.  I just don't think it is charter-ready yet, but will 
> NFSv4.2 is mentioned in the existing charter.  It is not in my charter draft, 
> On Thu, May 11, 2017 at 12:58 PM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 11 May 2017 20:17:18 -0500

> that Spencer S. suggested to allow later addition of milestones :-)

Traditionally, IETF WG charters include a lot of things that do not
have active WG documents at the time the charter is adopted.  On the
contrary, documents are generally not eligible to become WG
documents unless there is some existing item in the WG charter that
applies to the document being considered.

So, I would say it's fine and expected to have charter items that do
not have an accompanying WG document, or even charter items that
have no candidate individual documents, if there is consensus that
the WG should do work in that area.

-Ben

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 11 May 2017 20:32:58 -0500

> or RFC that discusses the requirements of a WG charter?

I'm no Spencer, but I believe BCP 25/RFC 2418 is the current
documentation in this regard.

-Ben

---

From: David Noveck <[email-masked]>
Date: Thu, 11 May 2017 22:32:37 -0400

> don't feel it is ready to promote without further discussion.

I wasn't suggesting that it move forward without further discussion.
I was suggesting that we could well the have necessary discussion
by IETF-99.

My understanding of the objection is to it being a standards-track
document so that it could move forward as a WG Experiental
document and so get a milestone..

I think this is right as a Proposed Standard but it isn't really worth
spending a lot of time arguing about the matter.  The
important facts are that there is an implementation of this
in the Linux client, and that it could help address two important
performance issues in RPC-over-RDMA Version One: invalidation
overhead and excessive use of reply chunks.   As a result, server
implementers that care about performance will implement it,
whether it is a Proposed Standard, an Experimental RFC, an
Informational RFC, a work-in-progress, or an expired individual
submission.

I think it is best for us to have the discussion and move it forward
with whatever status the working group can agree on.

> RPC-over-RDMA Version Two could become a large project.

It doesn't have to be.  As I recall, you proposed, with good
reason, a very small Version Two.

As far as I'm concerned Version Two, as defined in
draft-cel-rpcrdma-version-two
is essentially Version One Done Right.   It is designed so that it is easy
to
create an implementation that interoperates with Version One peers.  The
changes are small and limited to:

   - A larger default inline threshhold.
   - A simple mechanism to exchange information about transport properties
   - The ability to define OPTIONAL extensions,
   - A more flexible remote invalidation scheme

> to construct prototypes?

I think the only way to find out is to ask people and the best way
to do that is to discuss the issue of moving this forward.  If  we had
that discussion and the working group was supportive, I would
probably look, when I implemented a Version One server, how
much work it would take to turn into a Version Two

I think we should just let the working group decide this issue.  if we
can't do that by IETF99, we might be able to do so by the Fall
Bakeathon.  I don't see what kind of consideration you are anticipating
having before proceeding to have the working group discussion.
server.   At that point i would have a better idea of

> Do we understand and agree on

No we don't but that is the reason for an extensible protocol.
We need that sort of agreement when we decide on what
extensions to develop.  I agree that that decision is a ways off.

> Have we reached the end of RoRV1's ability to deliver

Probably not.  There is a lot of implantation work to be done, but if
people do find bottlenecks that require an extension, it is better to be
prepared.  The reason for Version Two is to be able to make limited
changes when necessary.  It will not make performance improvements
on its own, except perhaps in environments that could benefit from the
more general remote invalidation support.

> while.

I recall haring claims that a big Version Two is needed but I
haven't heard them for a while.  If there are still people
advocating a large/massive Version Two, we can have the
necessary discussion as part of proposing advancing our
minimal Versiion Two.

> The answer shifts depending on whether we think

I don't think that's the case.  I think the question of
the out-of-band mechanism affects the urgency of
proceeding with a minimal Version Two.

Put another way, if cm-pm-msg had never existed, the
Version Two would be an urgent matter to be addressed
immediately.  Given that it does exist, I think we can take
a more leisurely approach and look to make this decision
in the next six months.

> the disposition of cm-pvt-msg.

I don't see the need but I think the clarity is availble.  The formalites
don;t really matter.  What matters is running code and it appears that
the running code provides the mechanism you are loking for, and just
eined an implemented.  I on't see how that affects the scopt Version Two,
however.  what is in rpcrdma-vrsion-two seems a goo fit for our current
need in any case.

> I'm struggling with an existential question.

I'll discuss the issue of performance being explicitly mentioned in the
charter
tomorrow.

On Thu, May 11, 2017 at 4:53 PM, Chuck Lever <[email-masked]> wrote:

> is submitted, then they should included.
>
> > I'm not sure we would be able to include things that aren't working
> group items at charter submission, although i believe that we would be able
> to take advantage of the provision that Spencer S. suggested to allow later
> addition of milestones :-)
> >
> > This has no current WG item.   I expect there would be some discussion
> of  possible paths forward in this area once migration-issues-13 is
> > For these there is no current WG document or I-d.  I am looking forward
> to work being done on these but don't expect these to be included in the
> charter submission.
> >
> >  - flexfile layout type
> >
> > I did ask Tom privately about setting a target date for this since it is
> a WG item.  I anticipate correcting this at IETF99 by guessing at dates
> until Tom nods his head.
> >
> and the work is well-scoped.
> >
> > True but I used having a WG document started or at least agreed upon as
> the gating criterion.  I'm willing to change that if you suggest an
> alternative criterion.
> >
> > > This has the following changes from the initial draft.  If you object
> to any of these changes, let me know right away:
> > >       • Deletion (for now) of the section NFSv4 Multi-Domain Access
> for FedFS.  Unlike the previously mentioned sections, this could come back,
> > >       • Added a reference within the Maintenance section, to the
> ability of technical updates to NFSv4 versions to include limited XDR
> changes.
> > >       • A start at a draft Milestones section (see below).  This is
> very limited since most WG documents are past WGLC right now.  I've made
> some reasonable assumptions regarding migration-issues.  When -13 is
> published, the way forward in this area will clearer and the list could be
> updated to include some expected documents to address the issues described
> in the Informational document.  BTW, I'm assuming there will be a need for
> WGLC for that document as a means of establishing consensus on a way
> forward for those issues even though I believe there is no point in
> publishing that document as an RFC.
> > > The NFSv4 protocol is designed to allow extension by the addition of
> new operations or new attributes, the creation of minor versions, and the
> > > Because the previous charter was at variance with the work the group
> was actually doing, the list of pending milestones that can determined now
> is quite limited.  To accommodate this situation and in light of the fact
> that maintenance activities are inherently unpredictable, new milestones
> that fall within the scope specified within the charter can be added after

---

From: David Noveck <[email-masked]>
Date: Fri, 12 May 2017 08:56:28 -0400

My understanding is that the objection is to it being a standards-track
document so that it could move forward as a WG Experiental
document and so get a milestone.

I think it is best for us to have the discussion and move the
document  forward with whatever status the working group
can agree on.

   - A larger default inline threshhold.
   - A simple mechanism to exchange information about transport properties
   - The ability to define OPTIONAL extensions
   - A more flexible remote invalidation scheme

I think the only way to find out is to ask people and the best way
to do that is to discuss the issue of moving this forward.  If  we had
that discussion and the working group was supportive, I would
probably look, when I implemented a Version One server, how
much work it would take to make it compatible with Version Two
as well.   I think that information might be helpful to people.

I think we should just let the working group decide this issue.  if we
can't do that by IETF99, we might be able to do so by the Fall
Bakeathon.  I don't see what kind of advance consideration you
are anticipating having before proceeding to have the working
group discussion.server.

> Have we reached the end of RoRV1's ability to deliver

Probably not.  There is a lot of implementation work to be done, but if

I don't see the need but I think the clarity is availble.  The formalites
don't really matter.  What matters is running code and it appears that
the running code provides the mechanism you are looking for, and it
has been and will be implemented.  I don't see how that affects the
scope of Version Two.  What is in rpcrdma-version-two seems a
good fit for our current needs in any case.

in a little bit.

On Thu, May 11, 2017 at 10:32 PM, David Noveck <[email-masked]> wrote:

>> > I did ask Tom privately about setting a target date for this since it
>> is a WG item.  I anticipate correcting this at IETF99 by guessing at dates
>> until Tom nods his head.
>> >
>> > > The following is the second iteration of my attempt to provide a
>> draft charter to be "whacked" (as Beepy put it) on the working group list
>> so all comments are appreciated.  However, if someone would like to present
>> an alternate draft, that's OK too.  We do need some sort of draft to work
>> from for a discussion in Prague.  Without it, a discussion in Prague is not

---

From: David Noveck <[email-masked]>
Date: Fri, 12 May 2017 09:54:58 -0400

> de facto focus of WG effort for the past five years.

Let's be clear on the context and stress that what I posted is my draft.  I
posted
it to facilitate discussion.  It is not posted on a this-is-the-answer
basis or a
take-it-or-leave-it basis.  It is posted on a suggest-a-change-or-an-
alternative basis.

> extending these protocols as needed."

I think we could get by with that but let me explain why I
had a specific section in my draft.

I think that a draft without this section would not give the IESG
a fair picture of the work we would be doing.  Let's consider
the likely agenda for our meeting at IETF99.

If you exclude the charter discussion, over half the minutes will
be taken up by performance-related sessions: 20 minutes for
you to "drone" :-), 40 minutes for Christoph's pNFS mapping types,
and probably 15 minutes devoted to trunking-related stuff.   This
is over 60% devoted to performance-related work.  Right now, it
is what we are doing, for good reasons.

A lot of these things will not have milestones in the draft we submit.
In that case we will have to add milestones afterwards.  it is likely
that Spencer D would approve those (as extensions) even if there
were not a specific performance section.  However, I am worried
that if there is no performance section in the charter, the IESG
might wonder about us straying from the agreed charter in devoting
so much effort to performance.

We might get pushback on the performance section from the IESG,
and if we get that we can drop it.  However I'd prefer us to explain now
the reasons we expect a considerable focus on performance over the
next few years.

   - For a long while, due to some fortunate circumstances, we have been
   behind the performance curve and have struggled to catch up.
   - Just as our efforts to do so have started to bear fruit, there is
   another likely burst of technological innovation, in the form of storage
   class memory, that will up the performance ante, requiring the WG to spend
   further efforts on the performance front.

> specifically about performance.

In fact, you just spent a while working on three of them.  Clearly,
the basic reason for RPC-over-RDMA is performance, even if the
title is not anything like the Trumpian "Really Huge NFS
Performance" :-)

> on the WG,

It is.

> then it seems to me that "performance" should
> not be part of our charter.

I don't see the logic here.  Extensions are also driven by
individuals in/on the WG, yet that is in the charter.  Putting
something in the charter does not imply that this will be
driven by someone other than the individuals in the WG.

> Improving performance should

It should be driven by people who have ideas and are willing to
explain and implement them, wherever they happen to live.  I
hope the IETF will facilitate that.

> I'm comfortable with the Maintenance and Extension sections,

Good.  It appears that Spencer S. is not comfortable
with the latter although he has not proposed an alternative.

> responsible for approving changes to certain sections of

I don't see the point of doing that. I don't expect the IESG to
care very much.  This is the kind of thing I would add if it
seemed that the charter was too short.

> better performance, and will continue working on it.

Good.  Your work so far has put us in a much better position
and I hope we can maintain the momentum.

> Still not sure whether that is something that needs to appear in
> the WG charter.

Is it fair to characterize your current position on this as "No Objection"?

If so, I'll leave the performance section in for now and we can resolve
this question at IETF99.

On Fri, May 12, 2017 at 8:56 AM, David Noveck <[email-masked]> wrote:

>> is essentially Version One Done Right.   It is designed so that it is
>> easy to
>>> > For these there is no current WG document or I-d.  I am looking
>>> forward to work being done on these but don't expect these to be included
>>> in the charter submission.
>>> >
>>> >  - flexfile layout type
>>> >
>>> > True but I used having a WG document started or at least agreed upon
>>> as the gating criterion.  I'm willing to change that if you suggest an
>>> alternative criterion.
>>> >
>>> > > This has the following changes from the initial draft.  If you
>>> object to any of these changes, let me know right away:

---

From: "Adamson, Andy" <[email-masked]>
Date: Fri, 12 May 2017 16:35:42 +0000

________________________________
From: nfsv4 <[email-masked]> on behalf of David Noveck <[email-masked]>
Sent: Thursday, May 11, 2017 7:56 AM
To: [email-masked]; [email-masked]
Cc: Spencer Dawkins at IETF
Subject: [nfsv4] New draft for working group charter

  *   Deletion of References to FedFS, per Chuck's comments.
  *   Inclusion of other ONC components in the Maintenance section, per Chuck's comments.
  *   Deletion of the out-of-date sections RFC5664bis and NFSv4.2.
  *   Deletion (for now) of the section NFSv4 Multi-Domain Access for FedFS.  Unlike the previously mentioned sections, this could come back, probably in the form NFSv4 Multi-Domain Access if someone (e.g. Andy) provides a draft charter section.

I don't think there is enough interest at present to add a multi-domain access section to the working group charter. The protocol piece that I think is missing is a cross domain authorization information piece such as is expressed for MIT Kerberos in this personal draft which has languished since June of 2011 due to non-interest.

A Generalized PAC for Kerberos V5

draft-sorce-krbwg-general-pac-02

Abstract

   This draft proposes a generalized authorization structure for the
   Kerberos V5 protocol.  Such an authorization structure would allow
   for greater interoperability among directory services and other
   related Kerberos services across differing realms.

--> Andy

  *   Added a reference within the Maintenance section, to the ability of technical updates to NFSv4 versions to include limited XDR changes.
  *   A start at a draft Milestones section (see below).  This is very limited since most WG documents are past WGLC right now.  I've made some reasonable assumptions regarding migration-issues.  When -13 is published, the way forward in this area will clearer and the list could be updated to include some expected documents to address the issues described in the Informational document.  BTW, I'm assuming there will be a need for WGLC for that document as a means of establishing consensus on a way forward for those issues even though I believe there is no point in publishing that document as an RFC.
  *   In order to help deal with our  limited set of current milestones, I've followed Spencer's suggestion and adapted the material he cited from the TCPM charter.

Maintenance

Extension

Performance Challenges

Date               Milestone

---

From: Chuck Lever <[email-masked]>
Date: Fri, 12 May 2017 12:38:33 -0400

> On May 12, 2017, at 9:54 AM, David Noveck <[email-masked]> wrote:
> 
> 
> Let's be clear on the context and stress that what I posted is my draft.  I posted
> it to facilitate discussion.  It is not posted on a this-is-the-answer basis or a 
> a fair picture of the work we would be doing.

In that case, how about instead of a section dedicated to
just performance, there is a section that summarizes all
areas where WG members are focusing, which includes
enabling the performance attributes of advanced network
fabrics and new storage technologies, enabling use of NFS
in warehouse virtualization environments, helping NFS adapt
to increasing network security challenges, and extending
the life of legacy storage by enabling pNFS access to it?

> Let's consider 
> 	• For a long while, due to some fortunate circumstances, we have been behind the performance curve and have struggled to catch up.
> 	• Just as our efforts to do so have started to bear fruit, there is another likely burst of technological innovation, in the form of storage class memory, that will up the performance ante, requiring the WG to spend further efforts on the performance front.

Yes, NFS/RDMA is a performance play, but IMO performance
is orthogonal here.

The WG's role in the RPC-over-RDMA Version One work was
clarification to produce refreshed standards. The WG is
responsible for ensuring correct interoperation. The WG
did not do any implementation or measurement, and did not
ask for a performance-related justification before I
started this work.

We'll have to agree to disagree on this.

> driven by someone other than the individuals in the WG.

Extension is a means to an end that is mediated by WG
actions. There can't be a protocol extension without
interaction with the WG.

Performance work is as much about implementation as it
is about protocol. Performance needs are discovered
outside the WG, and that is also where new protocol is
frequently invented. The WG's role in performance work
is IMO much more tenuous.

> seemed that the charter was too short.

It's not filler, IMO. It's almost less than a single
sentence. Who would approve change requests on those
registries if the WG were dissolved?

I don't see any harm in adding this, considering there
has been activity in this area since the charter was
last updated.

> Is it fair to characterize your current position on this as "No Objection"?

I think I'd like you to consider the expanded summary
of WG member activity I proposed above, instead of a
section that focuses solely on performance work.

> As far as I'm concerned Version Two, as defined in draft-cel-rpcrdma-version-two
> is essentially Version One Done Right.   It is designed so that it is easy to
> 	• A larger default inline threshhold.
> 	• A simple mechanism to exchange information about transport properties
> 	• The ability to define OPTIONAL extensions
> 	• A more flexible remote invalidation scheme

- There are significant improvements to error reporting.

- We need to figure out how to deal with one-way messages
in a protocol that uses request-grant credit management,
or drop the use of one-way messages.

- There needs to be new properties that describe transport
header size limits, chunk list length limits, segment
size limits, and so on.

- If V2 can take more time to mature, I would like to
consider ways to eliminate the need for reply size
estimation. I have a mechanism in mind that also can
reduce the frequency of Reply chunks.

- There are currently no V2 prototypes.

In other words, the scope of V2 has already stretched,
and what's more, it feels like more oak-barrel aging is
needed.

> I think we should just let the working group decide this issue.

Yes, I'd like more WG discussion. I'm simply not putting
a time limit on it because there seems to be a lot more
work to be done. Like you said above, if the WG reaches
some consensus about this later, it can be added to the
charter.

Or we can change the admittance criteria for the charter,
and let this in now, without a WG document. It seems to
me if the WG agrees to change the charter to include
some item, that is enough consensus to assume that there
will be some work done in that area. Perhaps having an
extant WG document is too high a bar?

> if we 

My point is that there is a limited pool of resources to
work on both implementations and protocol development.
Any work done on V2 means there is something that falls
off of some part of the V1 plate.

Now that the V1 documents are reaching maturity and
stability, many of us would like to focus on improving
our V1 implementations, and move ahead with the new
pNFS layout type ideas.

That doesn't mean V2 work can't happen, but I don't like
the idea of treating it like there's a giant vacuum that
has to be filled right now with new protocol work.

> proceeding with a minimal Version Two.

I think I said that.

> a more leisurely approach and look to make this decision

A more leisurely approach means we have room to consider
a wider variety of approaches for V2. Unless we explicitly
decide not to.

The tl;dr is there needs to be an explicit discussion
revisiting the scope of V2, after the disposition of
cm-pvt-msg is decided, if only to confirm that the
scope remains the same.

--
Chuck Lever

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 12 May 2017 16:48:14 +0000

David,

Please remove the milestone entries for everything except the migration issues work.  We have progressed far enough that our responsibility is to nudge the work through the rest of the process and by the time the charter is updated, we should be even further along.

I believe the charter should have two sections.  One for maintenance and one for potential areas of extended interest.  In that second section we can then include some of the examples of areas that have been an interest for those engaged with the group and appear to have growing energy.

From this we will then work with our AD on charter updates for the larger work items that deserve a notification to the IESG such that there is the potential for cross area/working-group touch points.  I will work with Spencer D. on this approach to see if it is workable from his point of view.

Spencer S.

From: nfsv4 [mailto:[email-masked]] On Behalf Of David Noveck
Sent: Thursday, May 11, 2017 4:57 AM
To: [email-masked]; [email-masked]
Cc: Spencer Dawkins at IETF <[email-masked]>
Subject: [nfsv4] New draft for working group charter

Maintenance

Extension

Performance Challenges

Date               Milestone

---

From: David Noveck <[email-masked]>
Date: Sat, 13 May 2017 07:28:36 -0400

> Yes, NFS/RDMA is a performance play, but IMO performance

Yes.  The motivation and the mechanism are orthogonal.

True.

> We'll have to agree to disagree on this.

I'm not sure exactly what we disagree about.

I guess we have to agree to disagree about whether we
disagree or not :-)

> actions. There can't be a protocol extension without

and the IESG and the RFC editor.

> is about protocol.

The part that is about implementation is generally not the
business of the WG.  However, there are parts that involve
protocol changes and the WG does need to be involved in those.

> Performance needs are discovered
> frequently invented.

Generally, needs are are discovered outside the WG, nnd
brought to the attentIon of the working group by one more
people authoring documents.  That applied to performnce
with Tom and Brent's RDMA work in the same way it applied
to the non-performance needs behind umask.

This applies to extensions, new transports like RPC-over-RDMA,
or new versions.  The WG is a means to an end since none
of these things can happen without the WG.

> The WG's role in performance work is IMO much more tenuous.

I don't see the difference.

> It's not filler, IMO.

You're right.  It isn't.  My point was that I wouldn't really expect its
absence
to be an issue for the IESG.

> It's almost less than a single
> sentence.

I think it would be exactly one sentence.

> Who would approve change requests on those

It would probably be TSWG, in that unlikely event.

OTOH, if this were not in the charter, I wouldn't expect us to be prevented
from doing this.  It wasn't in the previous charter.

> has been activity in this area since the charter was

I agree there will no harm.  I will put the sentence you want to see in the
*Maintenance* section of the next draft.

> section that focuses solely on performance work.

I can move in that direction (see below) but I think the
charter should not summarize *all *areas that we might work on.
There needs to be room for extensions not mentioned in the
charter.  I agree to not having a specific performance section,
but we might expand the *Extension* section to include a list
of the major areas that will drive extension work.

>From you list:

> fabrics and new storage technologies,

OK.

> enabling use of NFS in warehouse virtualization environments,

I don't object to this but I'm not clear what work is beIng referred to.
Please clarify.

> helping NFS adapt
> to increasing network security challenges,

OK.

> and extending

I"m not sure what this refering to either.

On Fri, May 12, 2017 at 12:38 PM, Chuck Lever <[email-masked]>
wrote:

> >       • For a long while, due to some fortunate circumstances, we have
> been behind the performance curve and have struggled to catch up.
> >       • Just as our efforts to do so have started to bear fruit, there
> is another likely burst of technological innovation, in the form of storage

---

From: David Noveck <[email-masked]>
Date: Sat, 13 May 2017 07:57:20 -0400

> Please remove the milestone entries for everything except the migration
issues work.  We have progressed far

> enough that our responsibility is to nudge the work through the rest of
the process and by the time the charter is

> updated, we should be even further along.

OK, but that leaves us with only one milestone :-(

OTOH, I think there could be additional milestones coming out of the
migration issues work that could be added at or before IETF99.  Stay tuned
for the publication of of migration-issues-13.

> I believe the charter should have two sections.

As a result of my discussions with Chuck, I will move toward a two-section
approach.

> One for maintenance

OK.

> and one for potential areas of extended interest.

I think there has to be some place for extensions in general, i.e. the kind
of thing we have been doing.

So I"m thinking about a larger extensions section that includes the
material you are talking about, but a more generic statement about
extensions (the stuff currently in the extensions section) as well.

> In that second section we can then include some of the examples of areas
that have been an interest for those

> engaged with the group and appear to have growing energy.

Yes, but we have to get WG agreement about what those are.

> From this we will then work with our AD on charter updates

I'm worried about "updates".  SInce we have had such trouble/delay doing a
single charter update, I'm worried about a program that foresees multiple
updates.

> for the larger work items that deserve a notification to the IESG such
that there is the potential for cross

> area/working-group touch points.

I'm not sure there are any of those.  What areas are you thinking of?

> I will work with Spencer D. on this approach to see if it is workable
from his point of view.

Since Spencer D is copied, it might be best if you filled in the details on
your proposal.  I expect Spencer would let us know if he thinks we are
barking up the wrong tree (or at the moon :-).

On Fri, May 12, 2017 at 12:48 PM, Spencer Shepler <[email-masked]>
wrote:

>
>
> David,
>
>
>
> Please remove the milestone entries for everything except the migration
> issues work.  We have progressed far enough that our responsibility is to
> nudge the work through the rest of the process and by the time the charter
> is updated, we should be even further along.
>
>
>
> I believe the charter should have two sections.  One for maintenance and
> one for potential areas of extended interest.  In that second section we
> can then include some of the examples of areas that have been an interest
> for those engaged with the group and appear to have growing energy.
>
>
>
> From this we will then work with our AD on charter updates for the larger
> work items that deserve a notification to the IESG such that there is the
> potential for cross area/working-group touch points.  I will work with
> Spencer D. on this approach to see if it is workable from his point of view.
>
>
> Spencer S.
>
>
>
> *From:* nfsv4 [mailto:[email-masked]] *On Behalf Of *David Noveck
> *Sent:* Thursday, May 11, 2017 4:57 AM
> *To:* [email-masked]; [email-masked]
> *Cc:* Spencer Dawkins at IETF <[email-masked]>
> *Subject:* [nfsv4] New draft for working group charter
>
>
>    FedFS.*  Unlike the previously mentioned sections, this could come
>    back, probably in the form *NFSv4 Multi-Domain Access* if someone
>    (e.g. Andy) provides a draft charter section.
>    - Added a reference within the *Maintenance *section, to the ability
> features. In the cases in which the required changes are inappropriate
> for the errata system, the working group will assist in publication of best

---

From: Chuck Lever <[email-masked]>
Date: Sat, 13 May 2017 12:35:19 -0400

> On May 13, 2017, at 7:28 AM, David Noveck <[email-masked]> wrote:
> 
> You're right.  It isn't.  My point was that I wouldn't really expect its absence 
> OTOH, if this were not in the charter, I wouldn't expect us to be prevented from doing this.  It wasn't in the previous charter.
> 
> Maintenance section of the next draft.
> 
> charter should not summarize all areas that we might work on.
> but we might expand the Extension section to include a list

NFSv4.2 features such as ALLOCATE and SEEK_HOLE and space
reservations and S2SC. These are very useful when preparing
or moving virtual disk images, or when thin provisioning is
in play.

The re-treading of NFS/RDMA. NFS/TCP is typically slow in
guests, but NFS/RDMA works in guests as well as NFS/TCP
works on bare metal.

> I"m not sure what this refering to either.

The pNFS flexfiles layout type. Feel free to adjust this
item if you feel my simple summary misses the mark.

> On Fri, May 12, 2017 at 12:38 PM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sat, 13 May 2017 16:08:17 -0400

> or moving virtual disk images, or when thin provisioning is

I get it now.  I think I was distracted by the word "warehouse".

The IESG could be as well.  How about:

Features to make NFS more suitable for use in virtualized

environments.

> guests, but NFS/RDMA works in guests as well as NFS/TCP

Cool.  it seems to me that there is room for improvement in that
result.  That would primarily be implementation work, but the
protocol might be able to help as well.

> item if you feel my simple summary misses the mark.

It is not that you missed the mark.  It is just when i saw
"legacy storage", I thought "spinning metal".

I'm thinking that a better way to say this might be:

Provide NFSv4 to files stored using other file access

protocol, through the definiton of appropriate pNFS mapping
types.

On Sat, May 13, 2017 at 12:35 PM, Chuck Lever <[email-masked]>
wrote:

> > OTOH, if this were not in the charter, I wouldn't expect us to be
> prevented from doing this.  It wasn't in the previous charter.
> >

---

From: Rick Macklem <[email-masked]>
Date: Sat, 13 May 2017 20:39:07 +0000

David Noveck wrote:
[stuff snipped]
> protocol, through the definiton of appropriate pNFS mapping
I'm guessing that by "mapping types" you are referring to "layout types"?
If I'm guessing, maybe others would be too.

rick
[lots of stuff too hard to delete in this web mail system, so I'm leaving it here;-)
On Sat, May 13, 2017 at 12:35 PM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:

> On May 13, 2017, at 7:28 AM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
>

> On Fri, May 12, 2017 at 12:38 PM, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:
>
> > On May 12, 2017, at 9:54 AM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:
> >
> >
> > On Fri, May 12, 2017 at 8:56 AM, David Noveck <[email-masked]<mailto:[email-masked]>> wrote:

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Sun, 14 May 2017 10:57:17 -0400

> On May 13, 2017, at 4:08 PM, David Noveck <[email-masked]> wrote:
> 

That could mean running on my desktop. I'm specifically talking
about features that are designed to make administering a large
number of guests and hosts more straightforward.

How about:

 "features that facilitate administration of NFS storage in
  scale-out virtualization environments"

> protocol, through the definiton of appropriate pNFS mapping 

That seems like what all the pNFS layout types do.

The Abstract of the flexfiles I-D says:

   The flexible file layout type is defined in this
   document as an extension to pNFS which allows the use of storage
   devices in a fashion such that they require only a quite limited
   degree of interaction with the metadata server, using already
   existing protocols.  Client side mirroring is also added to provide
   replication of files.

"using already existing protocols" is not terribly helpful.
That's also what other pNFS layout types do.

Perhaps one of the Primary Data engineers can give us a
crisp one-sentence mission statement for flexfiles.

If not, then you could make a generic statement about
broadening the repertoire of layout types to include
new storage protocols such as NVMe on Fabrics, and
leave it at that. I'd hate to do that, though, since
a lot of work has gone into flexfiles.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sun, 14 May 2017 11:36:55 -0400

>  scale-out virtualization environments"

OK

> That seems like what all the pNFS layout types do.

Except for pNFS file.

> That's also what other pNFS layout types do.

I think the import is intended to be that no control
protocol is required.

> crisp one-sentence mission statement for flexfiles.

I think they will, although we might have to wait until IETF-99.

> a lot of work has gone into flexfiles.

A lot of work has gone into it and it is one of the few
working group documents that have not been through
WGLC.  Therefore, it has to be represented in the updated
charter, if possible. But I think I need Primary Data's input to
put something about it in the charter draft.

I think Christoph's NVMe work would be adequately represented
by your statement about use of new storage technologies.
You can judge how that works once I send raft #3 out.
We can add something flex-files-related once Primary
Data can suggest something appropriate to add.