Subject: [nfsv4] FW: New Version Notification for draft-andros-nfsv4-client-multipath-discovery-00.txt

From: "Adamson, Andy" <[email-masked]>
Date: Thu, 9 Feb 2017 21:07:05 +0000

FYI

On 2/9/17, 3:59 PM, "[email-masked]" <[email-masked]> wrote:

A new version of I-D, draft-andros-nfsv4-client-multipath-discovery-00.txt
has been successfully submitted by William A. (Andy) Adamson and posted to the
IETF repository.

Name:		draft-andros-nfsv4-client-multipath-discovery
Revision:	00
Title:		Trunking Discovery For Network File System Version 4.1
Document date:	2017-02-09
Group:		Individual Submission
Pages:		13
URL:            https://www.ietf.org/internet-drafts/draft-andros-nfsv4-client-multipath-discovery-00.txt
Status:         https://datatracker.ietf.org/doc/draft-andros-nfsv4-client-multipath-discovery/
Htmlized:       https://tools.ietf.org/html/draft-andros-nfsv4-client-multipath-discovery-00

Abstract:
   Connection trunking is the use of multiple transport connections to
   increase data and request throughput between one NFS client and
   server pair.  This document describes a means for an NFS version 4.1
   client to discover NFS version 4.1 server multipath addresses that
   may be used for connection trunking.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 9 Feb 2017 17:36:04 -0500

Hi Andy,

On 9 February 2017 at 16:07, Adamson, Andy <[email-masked]>
wrote:

> URL:            https://www.ietf.org/internet-
> drafts/draft-andros-nfsv4-client-multipath-discovery-00.txt
> Status:         https://datatracker.ietf.org/
> doc/draft-andros-nfsv4-client-multipath-discovery/
> Htmlized:       https://tools.ietf.org/html/d
> raft-andros-nfsv4-client-multipath-discovery-00
>
>
> Abstract:
Shouldn't there be a discussion of the role of DNS in a document that aims
to describe how to do transport discovery?
After all, DNS already has a mechanism for doing all that is described
here, and is already used for autoconfiguration by all the NFS sideband
protocols such as kerberos and ldap. Furthermore, both RFC5661 and RFC7530
explicitly state that the UTF-8 encoded strings returned in the
fs_locations and fs_locations_info attributes are allowed to contain DNS
host names instead of transport-specific addresses.

So, what are the advantages of obtaining the transport information using
this method, rather than directly looking up the DNS hostname? Why would I
want to trade off flexibility around how I implement my pseudofs or whether
or not I replicate it?

BTW: I'm curious about the paragraph in Section 3.3.1.2 that states

When creating an fs_locations_info pseudofs multipath replica list,
> the server SHOULD NOT set the FSLI4BX_GFLAGS, FSLI4BX_CLSIMUL,
> FSLI4BX_CLHANDLE, FSLI4BX_CLFILEID, FSLI4BX_CLWRITEVER,
> FSLI4BX_CLCHANGE, nor FSLI4BX_CLREADDIR fs_locations_server4 fls_info
> flag fields. The client MUST ignore these flags.

How does the client determine that it MUST ignore those flags? Can it
determine that this is a  pseudofs?

Cheers

  Trond

---

From: David Noveck <[email-masked]>
Date: Thu, 9 Feb 2017 20:17:13 -0500

>  Can it determine that this is a pseudofs?

I don't see how it can.

I think interrogation of the fs_locations might as well happen for all root
fs's, without depending on whether the root fs is a pseuo-fs or not.

If the root is a real (i.e non-pseudo) fs which is replicated the client
can determine whether a given address is or is not another path to the same
server, by checking if the other address has the same server name when
connecting. Any candidate addresses where the server name does not match
would have to be discarded in any case. This is apart from any
security-based check that the server is not an imposter giving you an
invalid server name.

On Thu, Feb 9, 2017 at 5:36 PM, Trond Myklebust <[email-masked]> wrote:

> How does the client determine that it MUST ignore those flags? Can it determine that this is a  pseudofs?
>
>
> Cheers
>
>   Trond
>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 10 Feb 2017 11:46:19 -0500

> On Feb 9, 2017, at 5:36 PM, Trond Myklebust <[email-masked]> wrote:
> 
> Hi Andy,
> 
> On 9 February 2017 at 16:07, Adamson, Andy <[email-masked]> wrote:
> FYI
> 
> Shouldn't there be a discussion of the role of DNS in a document that aims to describe how to do transport discovery?
> After all, DNS already has a mechanism for doing all that is described here, and is already used for autoconfiguration by all the NFS sideband protocols such as kerberos and ldap. Furthermore, both RFC5661 and RFC7530 explicitly state that the UTF-8 encoded strings returned in the fs_locations and fs_locations_info attributes are allowed to contain DNS host names instead of transport-specific addresses.
> 
> So, what are the advantages of obtaining the transport information using this method, rather than directly looking up the DNS hostname? Why would I want to trade off flexibility around how I implement my pseudofs or whether or not I replicate it?

All good questions.

Rather than advocating for an approach similar to RFC 6641, I
assume you are requesting that more rationale should appear in this
document about why we want to use an NFS operation for this purpose.

Typically rationale does not appear in protocol specifications, as
this document aspires to be. Is there WG interest in making this
document a problem statement rather than a protocol specification?
I wouldn't object to backing up a step, if that is desirable.

Anyway, such a rationale might look like this:

For FedFS domain roots, which rarely change, DNS is appropriate.
We feel that it is important to enable servers to control their
own multi-path lists and enable them to be adjusted dynamically.
Separating the construction and publication of these lists by
placing their publication in a different administrative realm is
cumbersome.

Potential use cases include:

1. Server has public and private interfaces. A client accessing a
public interface would not care about interfaces it cannot access,
and it would be a leak of information for the client to be told
about them. The server can control the content of the multi-path list
based on a policy.

2. Server network configuration can change dynamically (for example,
after a server reboot). An in-band NFS operation enables each server
to compose its multi-path lists automatically based on local policy
and other local information.

3. The mechanism needs to convey information about interface type
and speeds (as SMB multichannel does) to enable clients to make
robust choices about which path(s) to use. A client might want to
choose a 40Gbps RDMA-capable interface over a 56Gbps Ethernet
interface, for example. LDAP and Kerberos service discovery does
not have this requirement.

4. Server may want to actively balance client load across her
interfaces by adjusting the multi-path interface lists that are
passed to clients.

The long and short of it is we'd like to enable programmatic
capabilities during the composition of multi-path lists. A DNS-based
mechanism, by itself, would make that challenging.

Storing such lists on an LDAP server, which has a well-known API to
accept infrequent modifications to its content, is another possibility.
However the use of LDAP in FedFS has proven to be an obstacle to its
adoption, due to the complexity of the LDAP protocol and LDAP server
administration. In addition, there being so little LDAP expertise
within the IETF these days, constructing well-vetted LDAP-based
protocols tends to be a challenge.

> How does the client determine that it MUST ignore those flags? Can it determine that this is a  pseudofs?

We assumed an exported "/" is always a pseudofs. There doesn't seem
to be any good reason to replicate a pseudofs; a pseudofs is tied
inextricably to a distinct server. It simply didn't make sense to
us for anyone to want to replicate it.

If these assumptions are not valid, then one alternative is to extend
the NFSv4 GETATTR operation by adding a new attribute that returns a
multi-path list.

The benefits to that approach:

- No confusion with existing replication and migration mechanisms;
this would be a clean start rather than adapting an existing
protocol element that might not be fully suited to the task

- No need to send pathnames in the multi-path list; only hostnames
or interface addresses are needed for multi-path lists

- Can send explicit interface speed and transport capabilities
in such a list, rather than abstracting them into "priority" and
"rank" numbers

The down side is that alternative might exclude multi-path support
for NFSv4.0 and possibly NFSv4.1 deployments. Our desire was to
enable multi-path via conventions to avoid the need for protocol
changes.

Another possibility is to introduce a new pNFS layout type for this
purpose. That could make it difficult to use for clients that have
no existing support for pNFS, but would allow the ability to
construct multi-path lists containing a relatively unconstrained
variety of information, thanks to the opacity of layout information
to the NFS protocol, and because such a layout could be specified
and extended independently of NFSv4. Also, servers could actively
recall these layouts if their interface configuration were to be
updated.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 10 Feb 2017 12:31:38 -0500

> this document aspires to be.

If that is typical, it not the universal practice.  There are often
good reasons to explain why you are doing something.  For
example, both xattrs and umask explain why they are adding the
extensions they are.  If they didn't, the documents would
be hard to understand

> Is there WG interest in making this

Not on my part.

> I wouldn't object to backing up a step, if that is desirable.

I just can't understand why it would be desirable.  I guess the only
case would be if people find your explanation unconvincing..  In any case,
I don't see the point of a separate document.  I would suggest
adding your explanatory text to the -01 and continuing the
discussion in the hope that we can reach agreement about this.

On Fri, Feb 10, 2017 at 11:46 AM, Chuck Lever <[email-masked]>
wrote:

> > Shouldn't there be a discussion of the role of DNS in a document that
> aims to describe how to do transport discovery?

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 10 Feb 2017 13:13:18 -0500

Hi Chuck,

> I wouldn't object to backing up a step, if that is desirable.
Yes, I think there needs to be a problem statement to accompany the
protocol spec, and justify the choices that are being made. Whether you do
it in the RFC or have a separate document is up to the WG.

> about them. The server can control the content of the multi-path list

Presumably you would need to have a VLAN or some such constraint on the
network topology anyway, no? Why wouldn't that VLAN also be able to enforce
the policy on not sharing the information?

> and other local information.

I can see how it may be safe to add interfaces during a server reboot, but
removing them would be somewhat obnoxious, as the clients may no longer be
able to pick up new lists.

This would be one of the downsides of having the server itself advertise
the list of interfaces; if the list changes in such a way that the
interface(s) used by the client is gone, then so has the client's ability
to update its list.

> not have this requirement.
Should this be a requirement? Network performance is typically something
that needs to be measured, and monitored. As you point out in (2), network
configurations (and topologies) can and often do change dynamically, as
does the load.

> interfaces by adjusting the multi-path interface lists that are

I can think of at least one major vendor that is shipping a product that
does dynamic load balancing of their server using DNS to route the NFS
clients to less congested NICs. It would therefore help if we could have a
discussion of the pros and cons of doing so in the problem statement.

> the NFSv4 GETATTR operation by adding a new attribute that returns a

I don't think you can assume that all servers will require a pseudofs. For
instance, if the server needs to export a single filesystem, then having a
pseudofs at the root would be quite redundant.

---

From: "Adamson, Andy" <[email-masked]>
Date: Fri, 10 Feb 2017 20:57:09 +0000

OK – I will add a Problem Statement section in draft-01.

-->Andy

From: nfsv4 <[email-masked]> on behalf of Trond Myklebust <[email-masked]>
Date: Friday, February 10, 2017 at 1:13 PM
To: Chuck Lever <[email-masked]>
Cc: "Adamson, Andy" <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [nfsv4] New Version Notification for draft-andros-nfsv4-client-multipath-discovery-00.txt

Hi Chuck,

On 10 February 2017 at 11:46, Chuck Lever <[email-masked]<mailto:[email-masked]>> wrote:

> On Feb 9, 2017, at 5:36 PM, Trond Myklebust <[email-masked]<mailto:[email-masked]>> wrote:
>
> Hi Andy,
>
> On 9 February 2017 at 16:07, Adamson, Andy <[email-masked]<mailto:[email-masked]>> wrote:
> FYI
>
> On 2/9/17, 3:59 PM, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>> wrote:
>
>
All good questions.

Yes, I think there needs to be a problem statement to accompany the protocol spec, and justify the choices that are being made. Whether you do it in the RFC or have a separate document is up to the WG.

Presumably you would need to have a VLAN or some such constraint on the network topology anyway, no? Why wouldn't that VLAN also be able to enforce the policy on not sharing the information?

I can see how it may be safe to add interfaces during a server reboot, but removing them would be somewhat obnoxious, as the clients may no longer be able to pick up new lists.

This would be one of the downsides of having the server itself advertise the list of interfaces; if the list changes in such a way that the interface(s) used by the client is gone, then so has the client's ability to update its list.

Should this be a requirement? Network performance is typically something that needs to be measured, and monitored. As you point out in (2), network configurations (and topologies) can and often do change dynamically, as does the load.

I can think of at least one major vendor that is shipping a product that does dynamic load balancing of their server using DNS to route the NFS clients to less congested NICs. It would therefore help if we could have a discussion of the pros and cons of doing so in the problem statement.

I don't think you can assume that all servers will require a pseudofs. For instance, if the server needs to export a single filesystem, then having a pseudofs at the root would be quite redundant.

--
Chuck Lever