Subject: [nfsv4] Problem statement: Dealing with compromised NSDBs

From: Chuck Lever <[email-masked]>
Date: Tue, 9 Apr 2013 12:45:05 -0400

Hi-

The FedFS ADMIN RPC protocol provides a mechanism for provisioning NSDBs on remote fileservers. The operations it provides are FEDFS_SET_NSDB_PARAMS, FEDFS_GET_NSDB_PARAMS, and FEDFS_GET_LIMITED_NSDB_PARAMS.

FEDFS_SET_NSDB_PARAMS specifies the name of an NSDB and the security mode to use when connecting to this NSDB.  The fileserver connects to an NSDB in order to resolve a FedFS junction.  The ADMIN protocol specification further says:

> On success, this operation returns FEDFS_OK.  When the operation
> returns, the new connection parameters SHOULD be used for all

  [ Btw: When would a fileserver choose not to use these connection
    parameters?  I think that SHOULD ought to be MUST, otherwise
    an administrator might specify a secure connection mode but get
    no security. ]

> subsequent LDAP connections to the given NSDB.  Existing connections
> MAY be terminated and re-established using the new connection
> parameters.  The connection parameters SHOULD be durable across
> fileserver reboots.

There are two security modes defined in the protocol specification:  FEDFS_SEC_NONE, which does not authenticate the LDAP server; and FEDFS_SEC_TLS, which uses START_TLS (RFC 4513) to authenticate the LDAP server.

When FEDFS_SEC_TLS is specified with the FEDFS_SET_NSDB_PARAMS operation, an x.509v3 certificate chain is provided to the fileserver, which it uses to authenticate subsequent connections to this NSDB.  FEDFS_SET_NSDB_PARAMS can change the connection security used by a fileserver to connect to a particular NSDB from NONE to TLS or TLS to NONE.

The idea is that over time, domain administrators will add NSDB connection parameters to each of their fileservers to allow FedFS junction resolution.  The specified NSDB may be the domain's own, or it might be an NSDB in a remote domain.

Many junctions on multiple fileservers can be created that use a particular NSDB.  There is no way to find such junctions without an exhaustive search.  Since filesystem namespace topology can evolve arbitrarily over time, a recorded pathname of any junction cannot be relied upon.

Now suppose we have two FedFS domains: example.net and university.edu.  Suppose university.edu fileservers have a number of junctions that refer to locations maintained by example.net, and thus use example.net's NSDB.

One day Mallory compromises example.net's NSDB, but the domain administrator there is on a long vacation.  The administrator at university.net discovers the compromise immediately, but has no control over the remote NSDB and cannot create a fresh x.509 certificate or verify that the contents of the NSDB are unmolested.  The only choice is to find and remove every junction in the university.edu that contains the compromised NSDB.

If university.edu is using a good implementation of FedFS, the command line tools it provides might allow the administrator to simply visit each of its fileservers and mark the example.net NSDB as compromised.  Any junction resolution that attempts to use that NSDB would fail, but the junctions remain in place.  When example.net's administrator gets back from holiday and cleans up the mess, the university.edu administrator can then update each of her fileservers with fresh connection parameters for that NSDB.

However, none of this can be done remotely using the FedFS ADMIN protocol.  It does not have a mechanism for removing NSDB connection parameters or for fencing a compromised NSDB.  This gap seems apparent to the audience whenever I give a talk about FedFS.

Here are some naive possibilities for addressing this:

1.  Add another connection security mode called, say, FEDFS_SEC_FENCED or FEDFS_SEC_REVOKED.  Subsequent fileserver connections to the NSDB fail with FEDFS_ERR_NSDB_AUTH.

2.  Add a new ADMIN protocol operation that removes an NSDB from a fileserver's NSDB connection parameters database.  Fileserver connections to any NSDB not in the fileserver's NSDB connection parameters database fail with FEDFS_ERR_NSDB_PARAMS.

Analysis:

1. would require a smaller change to the ADMIN protocol specification, which is an advantage given that the document is now in the RFC editor queue.  It would be a matter of adding the new value in the XDR definition of FedFsConnectionSec, and a new paragraph in the FedFsNsdbParams description in chapter 4 which discusses the meaning of FEDFS_SEC_FENCED and any behavior requirements.

Changing an NSDB's connection parameters from FEDFS_SEC_REVOKED to FEDFS_SEC_TLS would require the domain administrator to re-present the NSDB's x.509 certificate to a fileserver.  Our preference might be to insist on a certificate as a safeguard, or to keep the old one as a convenience.

This setting might be a little confusing for NSDBs that used to be in FEDFS_SEC_NONE mode, which require no authentication, and thus FEDFS_ERR_NSDB_AUTH might be unexpected.  An alternative would be to add a specific error code, say, FEDFS_ERR_NSDB_FENCED, to report this security setting when a connection to this NSDB is attempted.

In order for this to work securely, the SHOULD in the quotation above will have to be changed to MUST.  Otherwise, a fileserver can choose to continue connecting to an NSDB that the administrator wants fenced.

2. would be a somewhat larger overhaul, which could be a problem for finishing the ratification of the ADMIN protocol specification.  It might be better to leave this kind of change for an ADMIN v2 protocol.

I'm not sure what kind of changes are permitted at this stage.  If we can't make any protocol changes, then our only choice might be to embark on ADMIN v2 if we deem this gap important.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 18 Apr 2013 14:42:42 +0000

Thanks for the write-up, Chuck, and for accepting that this appears to be a scenario that should be covered initially.

I agree that this is something we should look at correcting and from your description below, option 1) appears to be the least invasive and appropriate.

IMO any change of this sort at this point in the process will require a WG last call on the changes to allow reasonable review and to assure the IESG that the changes have reasonably considered.  I know this is painful but is likely the prudent approach.

To assist in moving this along as quickly as possible, who do you consider "must" reviewers (within the WG and external if needed)?

I will contact Martin to see if he has an alternate opinion.

Comments from anyone else?

Spencer

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Chuck Lever
Sent: Tuesday, April 9, 2013 9:45 AM
To: [email-masked] mailing list
Subject: [nfsv4] Problem statement: Dealing with compromised NSDBs

Hi-

Analysis:

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Chuck Lever <[email-masked]>
Date: Thu, 18 Apr 2013 09:03:14 -0700

> IMO any change of this sort at this point in the process will require a WG last call on the changes to allow reasonable review and to assure the IESG that the changes have reasonably considered.  I…

Actually I'm happy for some formal review.  We're still waiting behind 3530bis, so we have a window here.

> To assist in moving this along as quickly as possible, who do you consider "must" reviewers (within the WG and external if needed)?

As a first pass, how about Rob Thurlow, Andy Adamson, Nico Williams, and David Black?  I would like some LDAP security expertise as well, but I'll need some help finding that.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 23 Apr 2013 12:22:26 -0400

> I'm not sure what kind of changes are permitted at this stage.  If we can't make any protocol changes, then our only choice might be to embark on ADMIN v2 if we deem this gap important.

To further discussion, let me propose a change set for option 1.  Since the FedFS ADMIN protocol document is in the RFC Editor queue and thus, I suspect, no longer under our control, I will simply describe the changes rather than posting a diff.

Section 2 describes the XDR for the ADMIN protocol.  Modify the FedFsConnectionSec enum to include a FEDFS_SEC_FENCED enumerator with a value different from FEDFS_SEC_NONE and FEDFS_SEC_TLS.  Modify the FedFsStatus enum to include a FEDFS_ERR_NSDB_FENCED enumerator
with a unique value (can we use 33, and shift the following values by one?).

Introduce a paragraph in section 3 describing FEDFS_ERR_NSDB_FENCED as follows:

  FEDFS_ERR_NSDB_FENCED:  The fileserver was unable to establish a connection with
  the NSDB because its connection security setting is FEDFS_SEC_FENCED.

In the section 4 paragraph describing FedFsNsdbParams, add a new paragraph as follows:

  A value of FEDFS_SEC_FENCED indicates that any attempt to connect to the NSDB MUST
  fail with the error FEDFS_ERR_NSDB_FENCED.  In this case, the secData array will have
  a length of zero.

The description of the FEDFS_LOOKUP_JUNCTION (5.4) and FEDFS_LOOKUP_REPLICATION (5.7) operations should be updated to mention FEDFS_ERR_NSDB_FENCED as a possible error return.

In section 5.8.2, change the following sentence:

  When the operation returns, the new connection parameters SHOULD be used for all
  subsequent LDAP connections to the given NSDB.

To this:

  When the operation returns, the new connection parameters MUST be used for all
  subsequent LDAP connections to the given NSDB.

As currently stated, this sentence allows implementations wiggle room to ignore the security mode setting.  In particular, when we add support for a "fenced" security mode, it would allow implementations to continue using a fenced NSDB.