Subject: Re: Delegation callbacks proposal

From: Brent <[email-masked]>
Date: Mon, 10 Sep 2001 17:00:06 -0700

I'd like to get some more input into the problem of callbacks
through NATs and/or firewalls. In last week's posting, I presented
the problem and followed it with a proposed solution.  Though I should
have made it more obvious that there are really two issues:

	1. The problem

	2. The solution

Mike Eisler referred to some previous discussion on callbacks, but
nobody (other than Dave Noveck's original proposal) mentioned the
problem with NATs and firewalls - and the conflict with the WG charter
statement:

	"Improved access and good performance on the Internet."

Given that delegation is our brightest hope for good performance
on the Internet, we must make this protocol perform well across
NATs and firewalls.

I proposed a solution that would utilize the transport endpoint
or connection used for SETCLIENTID as a callback address. This
would work around the NAT problem, and the re-use of an outbound
TCP connection would address some of the firewall problems.

Mike Eisler suggested that an implementation using a single
UDP socket for outgoing traffic might bottleneck the socket.
My callback proposal doesn't require a single source socket,
only that the one used for SETCLIENTID be available for
callbacks.  Ric Werme pointed out that Tru64 NFS uses a
single port for outgoing traffic and doesn't have a problem.

Jim Rees pointed out that AFS (which uses a similar caching/callback
model) already works through NATs using bi-directional RPC.
He supported the "connection re-use" proposal, but suggested an option
that would allow continued use of the cb_client4 in the SETCLIENTID
args for clients that don't want/need to work through NATs

My preference is to stick with a single callback policy, i.e.
mandate the re-use of a connection or source endpoint for
callbacks unless there's some compelling need to retain
the client's ability to specify a callback address. Otherwise,
we just burden our implementations with unnecessary cruft and testing.

I don't think there's any doubt that we do have a problem: the callback
mechanism conflicts with the charter.  We need to reach some consensus
on a solution.  If not connection re-use, then what else will fix 
the problem ?

	Ta
		Brent

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 10 Sep 2001 20:21:56 -0500

> on a solution.  If not connection re-use, then what else will fix 

I am not convinced that this is a "problem" that we need to solve
today.  The re-use of the connection for the callback path is
something that would fit nicely into a minor version of the protocol.

The proposed solution is not new to the working group (As Mike E. has
provided reference to) and it is certainly not something new to the
authors of the RFC as evidenced by these sections of RFC3010:

   1.1.6.  Client Caching and Delegation

   8.3.  Blocking Locks
	"must not rely on a callback mechanism"

   9.2.  Delegation and Callbacks
        "Because callback RPCs may not work in all environments (due to
   	 firewalls, for example), correct protocol operation does not depend
   	 on them.  Preliminary testing of callback functionality by means of a
   	 CB_NULL procedure determines whether callbacks can be supported.  The
   	 CB_NULL procedure checks the continuity of the callback path.  A
   	 server makes a preliminary assessment of callback availability to a
   	 given client and avoids delegating responsibilities until it has
   	 determined that callbacks are supported."

   9.2.1.  Delegation Recovery
         o    The use of callbacks is not to be depended upon until the client
               has proven its ability to receive them.

Since RFC3010 is a Proposed Standard, it has received the normal IETF
review cycle and the issue of failure to have the callbacks work with
NAT or firewall present.

I am not saying that the proposed idea isn't worthwhile; I just don't
think that given the history of developing the protocol and the list
of things that need to be addressed for correctness that it is the
best time to change the callback mechanism.

-- 

- Spencer -

---

From: Brian Pawlowski <[email-masked]>
Date: Mon, 10 Sep 2001 18:40:08 -0700 (PDT)

I remember this being discussed in great detail over e-mail and in the
meetings around that time when Dave Noveck first presented the design.

We achieved consensus then that the design as it stands was acceptable.

I would like to move forward and get consensus that we push "reuse
of a connection for callbacks" into a minor version effort following
on the base protocol specification (RFC 3010 + current in progress
edits).

Comments?

beepy

---

From: Neil Brown <[email-masked]>
Date: Fri, 14 Sep 2001 21:14:07 +1000 (EST)

> have made it more obvious that there are really two issues:

You noted two problems:  
  1/ NATs (and "fog on the internet" in general). 
  2/ "there's no standard for netid strings".

I have nothing to add to the discussion about 1/.
But if 2/ is a real problem, then it should be addressed somehow.

Your solution is one way to address it, but not necessarily the only
one. 
The NFSv4 protocol has two situations where a network address needs
to be communicated.

There is this situation, which uses a netid and a "universal address",
and there is the fs_location attribute that uses a "server" name,
to be looked up in the DNS or using inet_addr(3).
I always thought it was unfortunate that you couldn't specify a port
number in the fs_location field, and thus have multiple servers running
with the one IP address, possibly serving different filesystems with
vastly different implementations (though I cannot actually find a
concrete use for this.  "automounters" come to mind, as they run an
NFS service on a non-standard port, but they don't make for a strong
argument). 

I don't have any answers, but I think it is worth noting that Brent
mentioned two problems, not just one.

> 	2. The solution

I note in your proposed solution the sentence:

"If a dropped connection needs to be re-established, then client or
 server must make a best effort to re-establish when needed."

I'm not sure how the server would re-establish a connection.
Connecting to the client is quite different to sending an RPC request
up the connection that was established by the client (but that is
ofcourse the point).
You could suggest that if the client used a connection oriented
protocol, then it should listen on the same port that it sent the
SETCLIENTID request on.  However that assumes that that makes sense in
a protocol-independant sense, and I'm not sure that it does.
As a (somewhat strained) example, imagine connecting to the server
over a Unix domain socket.  I don't think that the client's end of the
connection has a name that can be used in a "connect" request by the 
server.

Now it could be that we only ever expect NFSv4.0 to be used over IPv4
or IPv6, and only with TCP or UDP.
That may be a perfectly reasonable restriction, or it may not (I
really have no idea).

If it is, then maybe both the clientaddr and the fs_locations address
should be a utf8 (DNS name or dotted quad or IPv6 notation) and a
int32 port number.
If it is not, then I wonder if the definition of fs_location is
entirely appropriate.

Or it might be reasonable to drop the part about suggesting that the
server might try to re-establish the connection.

NeilBrown

---

From: Brent <[email-masked]>
Date: Fri, 14 Sep 2001 14:45:16 -0700

> NFS service on a non-standard port, but they don't make for a strong

It's true that NFS mount commands have a "port" option, and the
NFS URL (RFC 2224) allows a port to be specified with the server
hostname.  One could argue that the replica list should simply
be a list of URLs.  There are rare instances where there's multiple
NFS servers running on a host - though the default one normally lives
on the "well known" port 2049. Yes, it's unfortunate that fs_locations
doesn't provide a port option.

> connection has a name that can be used in a "connect" request by the

Right, I'm sure there are cases where the server is unable to
restore a connection.  The only hope then is that the client
detect a failed connection and establish a new one before the
server needs it for callbacks.

	Brent

---

From: Eric Werme USG <[email-masked]>
Date: Mon, 17 Sep 2001 11:08:59 -0400

I just wandered through rfc 3010 and its May update.  While I can find
discussion about short leases and long leases, and notes that the server
sets the lease time, I don't where it communicates that nor do I see notes
on what lease times are to be used or implemention hints.  What are people
using?

I occurred to me that NFS over TCP implementations generally close "idle"
connections.  Tru64's server checks once a minute for connections not
used in the previous five.  The client does something similar, but accelerates
things if there are a lot of connections open.  If lease hold times are
less than that, the connections will automatically stay open.  In addition,
we could define "active" on both client and server to include active leases.

Therefore, the only times the connection is not available for a callback is
when either end is rebooted and, well, I guess leases have to be short enough
to timeout before the server restarts and ends the grace period.  (Is that
right?  No statd any more!)

For UDP, I suppose the server could keep the client's port in stable storage
along with other state if it uses that approach.  I don't see a need yet
to get the clients a well known port for requests or the callback.  I'm not
very concerned about firewall/NAT issues with UDP as the IETF insists on using
transports with congestion control comparable to TCP.  (Translation: you will
use TCP over a public WAN.)  UDP will still be useful in trusted LANs.

	-Ric Werme

---

From: RJ Atkinson <[email-masked]>
Date: Mon, 17 Sep 2001 11:38:22 -0400

At 11:08 17/09/01, Eric Werme USG wrote:
>connections.  

This does NOT appear to be the case for the NFS/TCP implementations
that are deployed around here (Aside: we have no DEC/Compaq systems now, 
nor will we anytime soon).

The NFS/TCP implementations around here appear (from external view) 
to be using the long-standing TCP keep-alive mechanism.

Ran

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 17 Sep 2001 20:48:38 -0500

> on what lease times are to be used or implemention hints.  What are people

The server's lease period is communicated with the "lease_time"
attribute; client is responsible for retrieving it via GETATTR.

It is unclear to me that implementors have chosen a final value for
the lease period.  Something that we are still gaining experience with.

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 17 Sep 2001 20:49:22 -0500

> to be using the long-standing TCP keep-alive mechanism.

The Solaris client/server will close idle connections after a few
minutes (5 and 6 I believe).

-- 

- Spencer -

---

From: Guy Harris <[email-masked]>
Date: Mon, 17 Sep 2001 18:53:03 -0700 (PDT)

> minutes (5 and 6 I believe).

And the Data ONTAP server will close idle connections after 6 minutes.