Subject: RE: [nfsv4] Trunking proposal

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 19 Oct 2006 15:57:55 -0400

A variable number of aggregated streams is supported.  I don't even
think it is limited to 4 billion. I think the discussion so far may have
given that impression because the focus is on the (binary) equivalence
relation between two addresses that answers the question
does-this-IP-and-that-connect-to-same-server.  But what is relevant to
the number of aggregated streams is the number of elements in the
equivalence classes generated by that equivalence relation, and that
really isn't limited, certainly not to any small number (and here
anything under 4 billion counts as small :-)

Not sure what excatly you want to do as far as your second point.  We
can report priorities for use for different addresses and whether RDMA
capability is present.  Could you give me an example of the kind of
thing you want to do?

-----Original Message-----
From: Erblichs [mailto:[email-masked]] 
Sent: Thursday, October 19, 2006 3:53 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Trunking proposal

Dave Noveck, et al,

	Hopefully this isn't too off target, since I am not 
	completely up to date on this proposal.

	A co-workers trucking product was based on a quad fast ethernet
	from one system to another. Thus, the trucking involved
	the aggegation of 4 network streams.

	So, my first suggestion is to support a variable number
	of aggegated streams. It seems to me that only a dual
	stream trunking implementation is being proposed.

	Second, multiple (64) bit ids could partion a set of bits
	to identify special characteristics of the server or
	client. One possible item would be to use the 8 most
	significant bits to identify an area type id.

	Thank you,

		Mitchell Erblich
		----------------

"Noveck, Dave" wrote:
> 
> This note presents a proposal regarding trunking that is based in
> large part on material that I presented at the interim working group
> meeting, although there are some changes and additions that reflect
> discussion at that meeting and subsequently.  The pdf of the
> powerpoint that I used at the meeting is available at
> http://www.nfsv4-editor.org/06interim/files/trunking-noveck.pdf
> 
> The proposal is to address some issues with regard to the handling
> of sessions in v4.1, and in particular how trunking is dealt with.
> In v4.0, trunking is dealt with as a problem rather than an
opportunity.
> 
> The protocol is structured so that each server IP address is to be
> treated by the client as a separate server and one client accessing
> the server via two separate IP addresses are treated by the server
> as two separate clients.  RFC3530 enforces this latter by requiring
> that the server IP address be included in the nfs_client_id4.
> 
> BTW, although I'll continue to use the term nfs_client_id4 here for
> the structure that includes a variable-length opaque and a reboot
> instance id, there is a general feeling that this is confusing as
> hell and much too close to clientid4.  Suggestions for an alternate
> name are welcomed.  Note that we are free to pick an alternate
> name (or even an alternate structure) for this, since with sessions
> we are switching from SETCLIENTID to a new op, which is currently
> CREATE_CLIENTID, and the former will be mandatory to not implement.
> 
> Sessions is designed to allow trunking but to make this work
> effectively, we need a way to allow the client and the server to
> determine the actual situation.  That is, the server must be able
> to reliably determine whether two client IP's are from the same
> client and similarly the client must have a reliable way to
> determine if two server IP's terminate in the same server.  To
> deal with various forms of server clustering it is desirable also
> to know whether two IP addresses may be used for connections that
> are part of the same session.
> 
> The first of these is relatively simple to deal with.  The
> nfs_client_id4 id field provides a client identification value
> which is independent of client IP.  The problem which we need to
> deal with is that in RFC3530 and in the v4.1 drafts as well, it
> is not independent of server IP.
> 
> The specs (RFC3530 and current v4.1 drafts) include server IP in
> the nfs_client_id4 in order, they say, to prevent a  subsequent
> CREATE_CLIENTD/SETCLIENTID, from deleting the locks if issued through
> a different server IP.  In fact, this argument is bogus since the
> locks would not be deleted because the same verifier would be used
> if the client is the same.
> 
> So I'm proposing that this inclusion of the server IP in
nfs_client_id4
> be deleted in v4.1, so that the server can determine whether two
client
> IP's corresponding to the  same client and thus can be part of the
same
> session, for example.
> 
> Even though the argument in the spec is spurious, there is still a
> reason to have the client the client accessing the server over two
> different IP's until and unless the client  has a reliable way of
> determining whether two server IP addresses are in fact the same.
> When a client thinks that two server IP addresses correspond to
> different servers while the server thinks that there is a single
> client, there are all sorts of confusion that can occur.  For example,
> while reclaiming locks, if the client got its locks with one server
> and then does a RECLAIM_COMPLETE, reclaims to the other server (from
> the client's point of view) would be stopped.  For things to work
> properly,  the server and the client need to be able to determine
> when two IP addresses correspond to the same server or client.
> 
> So since the client can identify itself using the nfs_client_id4,
> once the server address is stripped out, the issue remains how to
> determine when two server IP addresses correspond to the same server.
> 
> The proposal is for the server to have an id of the same form and
> having the same sort of uniqueness characteristics as the
> nfs_client_id4, in other words a server id string.  The identity
> equivalence classes defined by this string canbe verified using data
> from GSSRPC and the shared secret that Mike has introduced.
> 
> So the proposal is to replace CREATE_CLIENTID with EXCHANGE_ID.  The
> client presents the nfs_client_id4 with id and verifier just as it
> does for SETCLIENTID or CREATE_CLIENTID.  The server responds with
> a shorthand 64-bit clientid (although there has been some discussion
> of making this 128 bits.  Note that this need not be the same sort of
> quantity currently sent as the clientid in v4.0 requests.  That will
> be zero since the session in existing requests since the session
implies
> 
> the current client) and two other items discussed below.  Note that a
> new clientid is generated when the nfs_client_id4-verifier pair is
new,
> but that a CREATE_SESSION is required to confirm establishment of a
new
> set of locking state and deletion of locks associated with the old
> lease.
> 
> The second item provided is a server id structure.  Its most important
> field is an opaque id string with uniqueness characteristics like the
> id string in the nfs_client_id4
> 
> struct nfs_server_id4 {
>     opaque   server_major<>;
>     uint64_t session_minor;
> };
> 
> When two server IP's have the same server_major id the client treats
> the two IP's as connected to the same server and uses a common
clientid
> to identify itself.  The session_minor field allows the server to
> control binding of IP's to sessions.  When two IP's have a matching
> server_major and session_minor, the client, may bind connections to
> those two IP's a in a common session, while when the session_minor
> is different, it will create a new session to an existing clientid.
> 
> The third item returned by EXCHANGE_ID is an opaque string that
> defines the clientid scope.  This is to address issue 6, the need
> for a procedure to determine when clientid's used issued by one
> server may validly be recognized by another.  This is to allow
> transfer of state to occur as part of migration (see section
> 8.14.1 of RFC3530 and section 10.6.6 of draft-07).
> 
> As Trond has suggested for V4.0, a RENEW might be used to see if
> a server has state corresponding to a given stateid.  The equivalent
> for v4.1 is to establish a session using a previously foreign
> clientid.  The issue in both cases is how to eliminate the case
> that two non-cooperating servers have simply assigned the
> same clientid by accident.
> 
> By associating such a scope with a clientid, we make it possible
> to direct such "foreign" clientid's only when the scope matches.
> When the scope is different, there can be no state transfer, while
> when the scope is the same, servers have to co-ordinate their
> clientid's at least to the point that they can simply reject
> those that represent sets of state that they don't know about.
> 
> Simple servers that do their own clientid assignment would simply
> use the server_major field as their clientid scope, while groups of
> servers that co-ordinated their clientid assignment would have to
> have a procedure to pick one of their server_major id's to use that
> as the clientid scope.
> 
> So to summarize how trunking will work, let us consider a client
> accessing a server over a new IP address.  It does an EXCHANGE_ID
> sending its nfs_client_id4, which is the same it has sent to all
> other servers that it has communicated with.  It gets back a
> clientid and an nfs_server_id4.  This can then be compared to
> all of the existing nfs_server_id4's it has seen.
> 
> If it finds one that matches completely, then we have a connection
that
> can be bound to an existing session and we do a BIND_CONN_TO_SESSION
> to that session.
> 
> If we find a partial match, where the server_major matches but not the
> session minor, we will establish a new session for the existing
> clientid,
> via a CREATE_SESSION.  Note that if we have an erroneous collision of
> nfs_server_id4's then this connection bind will be rejected due to
> shared secret checking.
> 
> Note that in either of these two cases, the identity of the two
servers
> can be verified using the GSSRPC server principal.  If an RPCSEC_GSS
> context is created to server A and then to server B, the format of the
> server principal is nfs@FQDN.  Thus if two contexts to have the same
> target GSS principal name (nfs@FQDN) then the two servers are sharing
> cryptographic keying information. Thus they are the same server.
> 
> Note that we have reliable check on the associations indicated by the
> server_id, when we are either using RPCSEC_GSS, and when we are doing
> explicit connection binding, through the shared secret.  When neither
> of these apply the possibility exists that a inadvertent or malicious
> identity of nfs_server_id4 would allow two servers to be treated as
> the same when they are not.  Clients should consider carefully this
> possibility not using RPCSEC_GSS for EXCHANGE_ID and when dispensing
> with explicit connection binding.
> 
> If we have neither sort of server_id match, we create a new session
> for the new clientid and thus confirm it.
> 
> In this way, the client is aware of when server IP's are connected to
> the same server (using the server id and confirmation via RPCSEC_GSS)
> and when connections should share sessions (using the session_minor).
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Erblichs <[email-masked]>
Date: Thu, 19 Oct 2006 16:36:04 -0700

Dave Noveck,

	The area id within the server/client ids is analogous to
	to the area code before the local phone number digits.

	It could be used to identify whether the server/clients
	are local to each other.

	It could be used to identify whether the server/clients
	are gov related.

	It could be used to identify whether the connection
	should/could be handed to a mirrored server.

	Etc..

	I am not sure whether connection establishment COULD be
	modified, but a remote connection might implement some
	heuristics if the connection is thought to be a WAN or
	LAN type connection.

	Secondly, I would think that some local ids could be
	reused without a concern for id collisions if some
	type of "area id" or AS was used to identify the
	area/AS location of the server.

	Mitchell Erblich
	--------