Subject: Re: [nfsv4] back channel flags, CREATE_SESSION, BIND_CONN_TO_SESSION

From: "Matt W. Benjamin" <[email-masked]>
Date: Thu, 6 Oct 2011 11:11:15 -0400 (EDT)

Hi Trond,

The rationale for making it possible to arrive at a dedicated back channel for a session is, I think, mainly that it can yield greater throughput, a benefit which grows as workload increases.  Conversely, channel negotiation appears to not have many downsides.

My reasoning would be mainly that the dedicated configuration appears to minimise the throughput potentially lost to flow control or head-of-line blocking, in particular when the the underlying transport is TCP.  A slightly different way to look at it is, that it enables a simpler RPC dispatcher at both endpoints.  (I doubt the latter is more than a short-term consideration.  Still, having an ability to work around a class of transport/interop problems transparently, which the fallback strategy could permit, might be viewed as a win.)

If one is persuaded of the potential utility of a dedicated session back channel, it still might not appear clearly better to have the client initially propose a mixed channel and assume the server will object if it cares (as opposed to making a dedicated fore channel its initial proposal).  I think the fallback strategy is slightly better.  It offers both client and server input into the transport setup, and it has the benefit of being closest to what 41 servers currently do and the Linux client currently does.

As you correctly state, the client could proceed without a back channel in the presence of a "disagreeing" server and still be conformant with RFC 5661 (if perhaps not in spirit), but actually, I'm not certain that the Linux client actually does proceed as specified in this case, as presently implemented (3.1.0-rc8)--I think it may incorrectly assume that a mixed channel has been agreed on.

Back channel negotiation does what it does, seemingly, at very little cost in development (a few hundred lines, mostly structure formulas).  Maintenance costs are said to be proportional to code size (or so I have read, e.g., Page-Jones).  I'm not making light of this point, obviously everything has a cost.

Matt

----- "Trond Myklebust" <[email-masked]> wrote:

> On Wed, 2011-10-05 at 23:28 -0400, Trond Myklebust wrote: 
> > On Wed, 2011-10-05 at 19:21 -0400, Matt W. Benjamin wrote: 
> > > Currently, the Linux and I believe also the CITI Windows client
> always propose channels in both directions.  The Linux mainline Linux
> client doesn't know how to BIND_CONN_TO_SESSION, so trivially it won't
> negotiate any back channel if a server didn't agree to both directions
> today, either.  I've experimentally implemented a "fallback" model in
> a Linux client and (partly in a) Ganesha server.  I'd appreciate any
> feedback on the idea.
> > 
> > Yep. As I said, why should we bother adding support for servers
> that
> > don't? I can function perfectly well without pNFS support or
> delegation
> > support in such a case. Performance will suck, but why do I care?
> 
> To put it in more basic terms: what you are proposing will add
> development costs to the client and and an extra code burden to
> maintain
> long term. So what is in it for me?
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> [email-masked]
> www.netapp.com

-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Myklebust, Trond" <[email-masked]>
Date: Thu, 6 Oct 2011 10:29:51 -0700

> -----Original Message-----
> From: Matt W. Benjamin [mailto:[email-masked]]
> Sent: Thursday, October 06, 2011 11:11 AM
> To: Myklebust, Trond
> Cc: nfsv4; linux-nfs; nfs-ganesha-devel
> Subject: Re: [nfsv4] back channel flags, CREATE_SESSION,
> BIND_CONN_TO_SESSION
> 
> Hi Trond,
> 
> The rationale for making it possible to arrive at a dedicated back channel for a
> session is, I think, mainly that it can yield greater throughput, a benefit which
> grows as workload increases.  Conversely, channel negotiation appears to
> not have many downsides.

Why do we need such a huge volume of callbacks? They should be the exception rather than the rule, since their effect is always to slow down performance. Throughput is in addition limited by the number of session slots (currently 1).

> My reasoning would be mainly that the dedicated configuration appears to
> minimise the throughput potentially lost to flow control or head-of-line
> blocking, in particular when the the underlying transport is TCP.  A slightly
> different way to look at it is, that it enables a simpler RPC dispatcher at both
> endpoints.  (I doubt the latter is more than a short-term consideration.  Still,
> having an ability to work around a class of transport/interop problems
> transparently, which the fallback strategy could permit, might be viewed as a
> win.)

For whom? We have already implemented a callback model that is working fine for us. I have yet to see any evidence of the callback scalability breakdown scenario that you claim as a motivation. What I have (frequently) seen is scalability issues due to clients and servers running out of free TCP ports.

> If one is persuaded of the potential utility of a dedicated session back
> channel, it still might not appear clearly better to have the client initially
> propose a mixed channel and assume the server will object if it cares (as
> opposed to making a dedicated fore channel its initial proposal).  I think the
> fallback strategy is slightly better.  It offers both client and server input into
> the transport setup, and it has the benefit of being closest to what 41 servers
> currently do and the Linux client currently does.
>
> As you correctly state, the client could proceed without a back channel in the
> presence of a "disagreeing" server and still be conformant with RFC 5661 (if
> perhaps not in spirit), but actually, I'm not certain that the Linux client actually
> does proceed as specified in this case, as presently implemented (3.1.0-rc8)--
> I think it may incorrectly assume that a mixed channel has been agreed on.

I agree that we should fix that bug. I disagree that we need a second callback model to do so. Just turn off pNFS (and possibly WANTs if we ever implement them), and we should be fine.

> Back channel negotiation does what it does, seemingly, at very little cost in
> development (a few hundred lines, mostly structure formulas).
> Maintenance costs are said to be proportional to code size (or so I have read,
> e.g., Page-Jones).  I'm not making light of this point, obviously everything has
> a cost.

My objection is to the lack of a consensus on a single system for implementing features. I'm tired of being told that the spec allows you to do the same thing in 10 different ways, with an expectation that we should implement all 10 ways.
If we can't find a single good way of implementing a feature, then my preference is to drop support for that feature altogether (or to choose one implementation and to stick with it). My expectation for a standard is that it should aim to _reduce_ the number of different implementations so that we can concentrate our development and testing efforts. (BTW: pNFS is definitely not beyond criticism in this respect.)

IOW: I can see valid reasons for why we should test the case where the server refuses a mixed fore+back channel, but I don't see that as a reason to implement a second back channel model. That requires us to add code + tests (and perform regular regression tests) for that back channel mode, as well as dealing with the case where that model of operation too is rejected by the server.

Trond

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Thu, 6 Oct 2011 16:12:46 -0400 (EDT)

Hi Trond,

> For whom? We have already implemented a callback model that is working
> fine for us. I have yet to see any evidence of the callback
> scalability breakdown scenario that you claim as a motivation. What I
> have (frequently) seen is scalability issues due to clients and
> servers running out of free TCP ports.

For whom?  Well, I work primarily on servers.  I have worked experimentally on the Linux v41 client.  I am pretty sure I wasn't speaking of a "callback scalability breakdown."  I was discussing a tradeoff of ports for reduced overhead of flow control, lock contention, etc.  I wasn't trying to start an inflated theory discussion about these, I don't feel fully at ease doing so.  Still, I don't intuit that potential for port exhaustion trumps other considerations, it is one consideration among several.  I also mentioned some potential benefit for server implementations, such as the one we have been collaborating on.  I'd like to maximize its potential for success.

----- "Trond Myklebust" <[email-masked]> wrote:
>
> My objection is to the lack of a consensus on a single system for
> implementing features. I'm tired of being told that the spec allows
> you to do the same thing in 10 different ways, with an expectation
> that we should implement all 10 ways.
> If we can't find a single good way of implementing a feature, then my
> preference is to drop support for that feature altogether (or to
> choose one implementation and to stick with it). My expectation for a
> standard is that it should aim to _reduce_ the number of different
> implementations so that we can concentrate our development and testing
> efforts. (BTW: pNFS is definitely not beyond criticism in this
> respect.)

You think the standard should influence, so as to reduce, the number of competing implementations?  Maybe you meant 'implementation choices for <x> feature.'

My expectation from NFSv4 is that there is some room for innovation and experimentation.  I don't think we can avoid this if we wish to compete effectively with ad hoc storage protocols.

>
> IOW: I can see valid reasons for why we should test the case where the
> server refuses a mixed fore+back channel, but I don't see that as a
> reason to implement a second back channel model. That requires us to
> add code + tests (and perform regular regression tests) for that back
> channel mode, as well as dealing with the case where that model of
> operation too is rejected by the server.

I find I need to keep reminding my self what "assume" does.  But I assumed that the 5661 language for negotiating a back channel was in the draft (standard) because it added value to someone, and perhaps was implemented in some environment.  If so, perhaps such a person will chime in with an opinion.

> Trond

Thanks,

Matt

-- 

Matt Benjamin

The Linux Box

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 06 Oct 2011 22:27:11 -0400

> You think the standard should influence, so as to reduce, the number of competing implementations?  Maybe you meant 'implementation choices for <x> feature.'

I meant to say exactly what I said above: that a standard should reduce
the number of _different_ ways to implement the exact same feature. Find
something that works, and then add it to the standard. Do not waste my
time by adding experimental features and then coming back 9 months after
the standard has been published and stating that 'I just discovered that
feature A is better implemented using method X rather than method Y'.
That's not a standard: it's an experiment in progress.

> My expectation from NFSv4 is that there is some room for innovation and experimentation.  I don't think we can avoid this if we wish to compete effectively with ad hoc storage protocols.

I'm not trying to compete with ad-hoc storage protocols: they can always
find a niche market to peddle their ideas. Once somebody starts talking
about a standard, then the main concept is that there is room for
cooperation in order to achieve mutual benefit. I see no benefit here.

> I find I need to keep reminding my self what "assume" does.  But I assumed that the 5661 language for negotiating a back channel was in the draft (standard) because it added value to someone, and p…

Then you need to communicate that value to me. I never signed an
agreement to implement all of RFC5661.

NetApp

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 6 Oct 2011 19:36:17 -0700

I'm surprised that apparently all the servers that have a backchannel
share permit the association of both channels with the same connection.

It won't surprise me if the issues of contention, head of line, and RPC
dispatch, etc. that Matt mention come to haunt us. In a general purpose
client, this might not matter, but I predict special purpose clients
optimized for the specific use cases will use dedicated connections for
each channel.

On Thu, October 6, 2011 7:27 pm, Trond Myklebust wrote:
>> For whom?  Well, I work primarily on servers.  I have worked
>> experimentally on the Linux v41 client.  I am pretty sure I wasn't
>> speaking of a "callback scalability breakdown."  I was discussing a
>> tradeoff of ports for reduced overhead of flow control, lock contention,
>> etc.  I wasn't trying to start an inflated theory discussion about
>> these, I don't feel fully at ease doing so.  Still, I don't intuit that
>> potential for port exhaustion trumps other considerations, it is one
>> consideration among several.  I also mentioned some potential benefit
>> for server implementations, such as the one we have been collaborating
>> on.  I'd like to maximize its potential for success.
>>
>> You think the standard should influence, so as to reduce, the number of
>> competing implementations?  Maybe you meant 'implementation choices for
>> <x> feature.'
>
>> My expectation from NFSv4 is that there is some room for innovation and
>> experimentation.  I don't think we can avoid this if we wish to compete
>> effectively with ad hoc storage protocols.
>
>> I find I need to keep reminding my self what "assume" does.  But I
>> assumed that the 5661 language for negotiating a back channel was in the
>> draft (standard) because it added value to someone, and perhaps was
>> implemented in some environment.  If so, perhaps such a person will
>> chime in with an opinion.
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/