Subject: [nfsv4] Trunking proposal

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 6 Oct 2006 08:31:12 -0400

This note presents a proposal regarding trunking that is based in
large part on material that I presented at the interim working group
meeting, although there are some changes and additions that reflect
discussion at that meeting and subsequently.  The pdf of the
powerpoint that I used at the meeting is available at
http://www.nfsv4-editor.org/06interim/files/trunking-noveck.pdf

The proposal is to address some issues with regard to the handling
of sessions in v4.1, and in particular how trunking is dealt with.  
In v4.0, trunking is dealt with as a problem rather than an opportunity.

The protocol is structured so that each server IP address is to be 
treated by the client as a separate server and one client accessing 
the server via two separate IP addresses are treated by the server
as two separate clients.  RFC3530 enforces this latter by requiring 
that the server IP address be included in the nfs_client_id4.

BTW, although I'll continue to use the term nfs_client_id4 here for 
the structure that includes a variable-length opaque and a reboot
instance id, there is a general feeling that this is confusing as
hell and much too close to clientid4.  Suggestions for an alternate
name are welcomed.  Note that we are free to pick an alternate
name (or even an alternate structure) for this, since with sessions
we are switching from SETCLIENTID to a new op, which is currently 
CREATE_CLIENTID, and the former will be mandatory to not implement.  

Sessions is designed to allow trunking but to make this work
effectively, we need a way to allow the client and the server to 
determine the actual situation.  That is, the server must be able 
to reliably determine whether two client IP's are from the same
client and similarly the client must have a reliable way to
determine if two server IP's terminate in the same server.  To 
deal with various forms of server clustering it is desirable also
to know whether two IP addresses may be used for connections that 
are part of the same session.

The first of these is relatively simple to deal with.  The 
nfs_client_id4 id field provides a client identification value 
which is independent of client IP.  The problem which we need to 
deal with is that in RFC3530 and in the v4.1 drafts as well, it 
is not independent of server IP.

The specs (RFC3530 and current v4.1 drafts) include server IP in 
the nfs_client_id4 in order, they say, to prevent a  subsequent 
CREATE_CLIENTD/SETCLIENTID, from deleting the locks if issued through 
a different server IP.  In fact, this argument is bogus since the 
locks would not be deleted because the same verifier would be used 
if the client is the same.

So I'm proposing that this inclusion of the server IP in nfs_client_id4 
be deleted in v4.1, so that the server can determine whether two client 
IP's corresponding to the  same client and thus can be part of the same
session, for example.

Even though the argument in the spec is spurious, there is still a 
reason to have the client the client accessing the server over two 
different IP's until and unless the client  has a reliable way of 
determining whether two server IP addresses are in fact the same.  
When a client thinks that two server IP addresses correspond to 
different servers while the server thinks that there is a single 
client, there are all sorts of confusion that can occur.  For example,
while reclaiming locks, if the client got its locks with one server 
and then does a RECLAIM_COMPLETE, reclaims to the other server (from 
the client's point of view) would be stopped.  For things to work 
properly,  the server and the client need to be able to determine 
when two IP addresses correspond to the same server or client.

So since the client can identify itself using the nfs_client_id4,
once the server address is stripped out, the issue remains how to 
determine when two server IP addresses correspond to the same server.

The proposal is for the server to have an id of the same form and 
having the same sort of uniqueness characteristics as the 
nfs_client_id4, in other words a server id string.  The identity
equivalence classes defined by this string canbe verified using data 
from GSSRPC and the shared secret that Mike has introduced.

So the proposal is to replace CREATE_CLIENTID with EXCHANGE_ID.  The 
client presents the nfs_client_id4 with id and verifier just as it 
does for SETCLIENTID or CREATE_CLIENTID.  The server responds with 
a shorthand 64-bit clientid (although there has been some discussion 
of making this 128 bits.  Note that this need not be the same sort of 
quantity currently sent as the clientid in v4.0 requests.  That will 
be zero since the session in existing requests since the session implies

the current client) and two other items discussed below.  Note that a 
new clientid is generated when the nfs_client_id4-verifier pair is new, 
but that a CREATE_SESSION is required to confirm establishment of a new 
set of locking state and deletion of locks associated with the old
lease.

The second item provided is a server id structure.  Its most important
field is an opaque id string with uniqueness characteristics like the 
id string in the nfs_client_id4

struct nfs_server_id4 {
    opaque   server_major<>;
    uint64_t session_minor;
};

When two server IP's have the same server_major id the client treats 
the two IP's as connected to the same server and uses a common clientid 
to identify itself.  The session_minor field allows the server to 
control binding of IP's to sessions.  When two IP's have a matching 
server_major and session_minor, the client, may bind connections to 
those two IP's a in a common session, while when the session_minor
is different, it will create a new session to an existing clientid.

The third item returned by EXCHANGE_ID is an opaque string that 
defines the clientid scope.  This is to address issue 6, the need 
for a procedure to determine when clientid's used issued by one
server may validly be recognized by another.  This is to allow
transfer of state to occur as part of migration (see section
8.14.1 of RFC3530 and section 10.6.6 of draft-07).

As Trond has suggested for V4.0, a RENEW might be used to see if
a server has state corresponding to a given stateid.  The equivalent
for v4.1 is to establish a session using a previously foreign
clientid.  The issue in both cases is how to eliminate the case
that two non-cooperating servers have simply assigned the
same clientid by accident.   

By associating such a scope with a clientid, we make it possible 
to direct such "foreign" clientid's only when the scope matches.
When the scope is different, there can be no state transfer, while
when the scope is the same, servers have to co-ordinate their
clientid's at least to the point that they can simply reject
those that represent sets of state that they don't know about.
  
Simple servers that do their own clientid assignment would simply 
use the server_major field as their clientid scope, while groups of 
servers that co-ordinated their clientid assignment would have to 
have a procedure to pick one of their server_major id's to use that 
as the clientid scope.

So to summarize how trunking will work, let us consider a client
accessing a server over a new IP address.  It does an EXCHANGE_ID 
sending its nfs_client_id4, which is the same it has sent to all 
other servers that it has communicated with.  It gets back a 
clientid and an nfs_server_id4.  This can then be compared to 
all of the existing nfs_server_id4's it has seen.

If it finds one that matches completely, then we have a connection that
can be bound to an existing session and we do a BIND_CONN_TO_SESSION
to that session.   

If we find a partial match, where the server_major matches but not the
session minor, we will establish a new session for the existing
clientid,
via a CREATE_SESSION.  Note that if we have an erroneous collision of 
nfs_server_id4's then this connection bind will be rejected due to
shared secret checking. 

Note that in either of these two cases, the identity of the two servers 
can be verified using the GSSRPC server principal.  If an RPCSEC_GSS 
context is created to server A and then to server B, the format of the 
server principal is nfs@FQDN.  Thus if two contexts to have the same 
target GSS principal name (nfs@FQDN) then the two servers are sharing 
cryptographic keying information. Thus they are the same server. 

Note that we have reliable check on the associations indicated by the
server_id, when we are either using RPCSEC_GSS, and when we are doing
explicit connection binding, through the shared secret.  When neither
of these apply the possibility exists that a inadvertent or malicious 
identity of nfs_server_id4 would allow two servers to be treated as 
the same when they are not.  Clients should consider carefully this 
possibility not using RPCSEC_GSS for EXCHANGE_ID and when dispensing
with explicit connection binding.

If we have neither sort of server_id match, we create a new session 
for the new clientid and thus confirm it.

In this way, the client is aware of when server IP's are connected to
the same server (using the server id and confirmation via RPCSEC_GSS)
and when connections should share sessions (using the session_minor).

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Erblichs <[email-masked]>
Date: Thu, 19 Oct 2006 12:52:55 -0700

Dave Noveck, et al,

	Hopefully this isn't too off target, since I am not 
	completely up to date on this proposal.

	A co-workers trucking product was based on a quad fast ethernet
	from one system to another. Thus, the trucking involved
	the aggegation of 4 network streams.

	So, my first suggestion is to support a variable number
	of aggegated streams. It seems to me that only a dual
	stream trunking implementation is being proposed.

	Second, multiple (64) bit ids could partion a set of bits
	to identify special characteristics of the server or
	client. One possible item would be to use the 8 most
	significant bits to identify an area type id.

	Thank you,

		Mitchell Erblich
		----------------