Subject: [nfsv4] CB_ATTR_CHANGED for Labeled NFS

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 18 Sep 2012 05:10:37 +0000

Trond was mentioning that some seLinux implementations of Labeled NFS
are surfacing and we discussed an issue with whether or not to implement
CB_ATTR_CHANGED.

Consider a fully implemented Labeled NFS server,  it does not need the
callback. If there is a change, then it will fence off all operations and
the client will learn pretty fast. As a matter of fact, it may learn before the
callback is issued.

If the client has to know about any potential label changes, then it can
issue a delegation on the file.

The interesting case, the one I claim needed the callback, is the label
aware server, the one that just records/retrieves labels and lets the
clients do all of the enforcing. The only way I claimed the client could
be informed was via the callback.

But I'm wrong, the client can again issue a delegation on the file and
the act of changing the label will cause the delegation to be recalled.
The client will be very aware that the label changed.

If the client does not get the delegation, then it continues to get access
to the contents until the next communication with the server for that file.
But that is no different than what happens in a network partition.

So what unique problem does CB_ATTR_CHANGED solve for us
in Labeled NFS that delegations do not solve?

If the answer is none, then I propose we pull this requirement from the
release.

---

From: "Black, David" <[email-masked]>
Date: Tue, 18 Sep 2012 09:04:19 -0400

Tom and Trond,

Not implementing the label change callback requires that all labeled NFS clients
take out delegations on all files that they open because clients should be coded
the same way for fully implemented labeled NFS servers and label-aware-only NFS
servers.  I think this applies even when blocks of a labeled file are retained
in cache, as there are label changes that require cache flush of files whose data
MUST NOT be in the client's cache after the label change.

Is that what's wanted?  At a minimum, it's rather heavyweight mechanism to
deploy for a rare (but important) case (label change underneath a client, and
I suspect the requirement on clients to take out delegations ubiquitously will
at least be a SHOULD, and there will be environments in which it'll be absolutely
required based on the security policy that goes with the labels.

Thanks,
--David

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Haynes, Tom
> Sent: Tuesday, September 18, 2012 1:11 AM
> To: nfsv4 list
> Subject: [nfsv4] CB_ATTR_CHANGED for Labeled NFS
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 18 Sep 2012 14:16:39 +0000

On Sep 18, 2012, at 8:04 AM, "Black, David" <[email-masked]<mailto:[email-masked]>> wrote:

Tom and Trond,

servers.

Why? The client could detect that the server is label-aware-only (which is a LFS)
and act differently. You are also proposing below that clients might have to
be required to issue delegations due to local security policies.

 I think this applies even when blocks of a labeled file are retained
in cache, as there are label changes that require cache flush of files whose data
MUST NOT be in the client's cache after the label change.

How strong do we have to make the label-aware-only NFS server to operate
in a mode that can never be secure?

The elephant in the room is that a non-Labeled NFS client can
access the server and the file in question as long as it wants to.

That is the minimum behavior we want for the Labeled NFS client - pretty
easy to match.

What can be done after that?

1) Client keeps what is in the cache until it contacts the server and is
told the label has changed. Cost? It might have to query the label
more if it cares about the label getting less restrictive.

2) Client establishes a delegation on the file and is contacted by the
server if the label has changed. Cost? You list some above.

3) Server uses the callback to inform the client after the fact. Cost?
We implement an additional callback for something that you describe
as rare but important.

I actually find the first to be acceptable. I understand others may not.

But I don't understand why we would want to add a redundant feature - the
spec is already complex enough. We would also have to add an additional
code path which means more opportunity for regressions to enter.

The requirement is to notify the client when its locally cached version of the
file (whether metadata or data) has changed and needs to be refreshed.

A delegation satisfies that requirement.

Thanks,
--David

---

From: "Black, David" <[email-masked]>
Date: Tue, 18 Sep 2012 10:33:54 -0400

> and act differently.

And it'll result in clients that may not implement both label-aware-server and
fully-implemented server modes, so the answer to whether a labeled NFS client
is secure when used with a labeled NFS server becomes a version of "it depends"
- that's not the best answer from a security architecture perspective.

> You are also proposing below that clients might have to

That's clearly necessary with a label-aware server that doesn't issue callbacks.
I'm not sure whether delegations would be needed with a fully-implemented server.

> access the server and the file in question as long as it wants to.

IMHO, that's a red herring - it's the security admin's responsibility to ensure
that such clients cannot talk to a label-aware NFS server and that they are
treated appropriately by a fully implemented server.

Beyond that, I think we're talking past each other on use cases.  IMHO, the
critical use case for label change is that the label becomes more restrictive,
and hence a client that is currently using the file needs to stop, NOW!!
(that's what the M in Mandatory Access Control means).

What will the SElinux client do by default?:
- Delegations?  On which files?
- Poll for changes? On which files? How often?

Thanks,
--David

From: Haynes, Tom [mailto:[email-masked]]
Sent: Tuesday, September 18, 2012 10:17 AM
To: Black, David
Cc: nfsv4 list
Subject: Re: [nfsv4] CB_ATTR_CHANGED for Labeled NFS

Tom and Trond,

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 18 Sep 2012 18:53:10 +0000

> Why? The client could detect that the server is label-aware-only (which is a LFS)

And it’ll result in clients that may not implement both label-aware-server and
fully-implemented server modes, so the answer to whether a labeled NFS client
is secure when used with a labeled NFS server becomes a version of “it depends”
- that’s not the best answer from a security architecture perspective.

> be required to issue delegations due to local security policies.

That’s clearly necessary with a label-aware server that doesn’t issue callbacks.
I’m not sure whether delegations would be needed with a fully-implemented server.

If the client is making modifications, it better have a write delegation established to prevent a label change
from throwing away any of its modifications.

> access the server and the file in question as long as it wants to.

IMHO, that’s a red herring - it’s the security admin’s responsibility to ensure
that such clients cannot talk to a label-aware NFS server and that they are
treated appropriately by a fully implemented server.

Beyond that, I think we’re talking past each other on use cases.  IMHO, the
critical use case for label change is that the label becomes more restrictive,
and hence a client that is currently using the file needs to stop, NOW!!

How does using delegations prevent that?

Show me one case where a delegation does not suffice and a callback does.

(that’s what the M in Mandatory Access Control means).

The 'M' in Mandatory Access Control means that both the client and server are fully Label NFS capable.

Thanks,
--David

From: Haynes, Tom [mailto:[email-masked]<http://netapp.com>]
Sent: Tuesday, September 18, 2012 10:17 AM
To: Black, David
Cc: nfsv4 list
Subject: Re: [nfsv4] CB_ATTR_CHANGED for Labeled NFS

Tom and Trond,

---

From: "Black, David" <[email-masked]>
Date: Tue, 18 Sep 2012 15:19:40 -0400

Summarizing ...

Do you and Trond expect an NFS client that cares about label changes to
hold a delegation on every file that it accesses?

If the answer is a long form of "Yes and that is a reasonable thing to do."
... then we're done here , and I'll go away ...

I agree that the client can get all the notifications that it needs via
delegation revocation, but I think they're rare enough that the callback
is a preferable mechanism courtesy of the overhead of having to take out
a delegation every time.

OTOH, if there's another reason that I've missed why a labeled NFS client
should always request a delegation, this discussion is moot.

Thanks,
--David

From: Haynes, Tom [mailto:[email-masked]]
Sent: Tuesday, September 18, 2012 2:53 PM
To: Black, David
Cc: nfsv4 list
Subject: Re: [nfsv4] CB_ATTR_CHANGED for Labeled NFS

Thanks,
--David

Tom and Trond,

---

From: "Myklebust, Trond" <[email-masked]>
Date: Tue, 18 Sep 2012 19:44:00 +0000

On Tue, 2012-09-18 at 15:19 -0400, Black, David wrote:
> If the answer is a long form of “Yes and that is a reasonable thing to
> do.”
> 
> ... then we’re done here , and I’ll go away ... 
> 
>  
> 
> delegation revocation, but I think they’re rare enough that the
> callback
> 
> OTOH, if there’s another reason that I’ve missed why a labeled NFS
> client
> 

In either case, the client needs to register each file for which it
wants a notification, and so I can't see how one mechanism is more or
less heavy than the other. In practice, I don't see how either model can
scale and so the likely result is that we will be forced to adopt a
polling model for label changes.

Yes, that means that client-enforced MAC models will be less secure than
server-enforced ones, but that shouldn't come as a surprise to anyone.

Cheers
  Trond

> On Sep 18, 2012, at 9:33 AM, "Black, David" <[email-masked]>
> wrote:
> 
> 
> 
> 
> If the client is making modifications, it better have a write
> delegation established to prevent a label change
> 
> 
> On Sep 18, 2012, at 8:04 AM, "Black, David" <[email-masked]>
> wrote:
> 
> 
> 
> 
> 
> Tom and Trond,
> 

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
[email-masked]
www.netapp.com

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 19 Sep 2012 16:29:17 +0000

Lets talk about implementation of the CB_ATTR_CHANGED.

It depends on the server storing state for each file opened by each client.

What happens if the server reboots? If the state is persistent, no problem.

If it is not, how does the server poll each client to see what is open?

Does the client have to detect the server reboot (which it will do anyway)
and then inform the server about files it has open? Does it do this with
another OPEN? Or does the fact that the client presents the FH and
a matching subject label really represent what we are after? (Which I
think it does for this discussion.)

In the reboot scenario, how long does the server not allow a client
to modify the security attribute of a file? Okay, somewhat moot, the
case that matters is if another client is writing and it better have
a WRITE delegation. For the read case, if the client tries to
register the fact it has the file open and has an insufficient
security label, then it would have to flush its local content.

What happens in a network partition?

Besides recovery, what happens if a client B changes the security
attribute and when the server tries to notify client A, it can not
reach it? Does the server have to keep retrying?

While much of this can be handled by a client actually probing/polling
for the security attribute, there are scenarios above where
the server may want to take no action for a small delta of time.

We've already tackled all of these issues in 3550 and 5661 for
locks and delegations.

Redoing it all for this one callback is Trond's assertion that
both approaches are heavy.

Hence, if one requires no effort to code, it is preferable.

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 19 Sep 2012 16:50:37 +0000

> server-enforced ones, but that shouldn't come as a surprise to anyone.

Actually, with the scenario David described, they are equal to each other
and perhaps even to a system enforcing policies locally.

The use case is that the client has cached contents and when the object
label is modified on the server, the client needs to start enforcing that
new label.

Without delegations, until the client attempts to access the file on the
server, the cache contents will be retained. That is regardless of whether
the server is enforcing or not.

How do current systems that decide NFSv3 access based on external
databases or network topology handle this today? I can't see them
being able to invalidate the remote caches.

And for a strictly local access,  if one process changes the object label,
does that invalidate all page caches, etc?

I.e., we will be just as good as current distributed MAC implementations
without callbacks or delegations, but we will be better with either approach.

---

From: "Noveck, David" <[email-masked]>
Date: Wed, 19 Sep 2012 12:53:17 -0400

I'd say the client has to do a reclaim-type OPEN within the grace period.  If he doesn't, he only has a stateid that he can't use any more and the file is effectively closed.  So he has lost his right to be notified of any label change.

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 19 Sep 2012 16:59:38 +0000

> It depends on the server storing state for each file opened by each client.

No. The whole problem is that it won't be limited to open files. Labels
affect all operations on a file/directory/symlink/... Everything from a
lookup or a readlink to a rename or unlink operation can be subject to a
MAC.

> the server may want to take no action for a small delta of time.

When do we expect a client-enforced Labeled NFS implementation to change
on the server? That basically means that the label is being changed by
some process that is not running on the client.

Given that the client is enforcing the MAC, there is no way in that
scenario to avoid races: if the label changes on the server after we've
checked the MAC, but before we issue the create operation (or whatever),
then we can't undo that operation or cancel it.

> Hence, if one requires no effort to code, it is preferable.

I'm not so much worried about coding it as I am about the server having
to track huge quantities of files.

NetApp

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 19 Sep 2012 17:05:16 +0000

> label is modified on the server, the client needs to start enforcing that

My point is that in order to have the client police operations when the
labels are changing on the server would require some kind of locking
scheme. Otherwise, you have to live with the imperfections of caching.

This is exactly why the NFS protocol has always had a server-enforced
security model.

> being able to invalidate the remote caches.

As I said above, the NFS model is server-enforced. If the acl changes,
then the effect on a rename or unlink RPC call is instant.

NetApp

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 19 Sep 2012 17:45:15 +0000

> then the effect on a rename or unlink RPC call is instant.

That said...

We can get very close to a server-enforced model by using the VERIFY
operation to prefix non-idempotent operations with a check that the
label hasn't changed. The problem with that approach is that
FATTR4_SEC_LABEL is a pretty beefy object, so doesn't really lend itself
well to use with VERIFY.
Could we perhaps add a change_attribute-like metadata object for this
purpose? I suspect that a server-generated 64-bit hash of the label
string might be sufficient.

NetApp

---

From: "Black, David" <[email-masked]>
Date: Wed, 19 Sep 2012 15:25:37 -0400

> some process that is not running on the client.

Revocation - the document has been misclassified at too low a level, allowing
too much access - the classification needs to be changed and all now-disallowed
usage of the document needs to be terminated ASAP with prejudice.

> then we can't undo that operation or cancel it.

That race condition is unavoidable, as in the revocation use case, the client
may already have (or have permitted an application to have) access that should
not have been granted in 20/20 hindsight.  Prompt action to prevent further
MAC-disallowed access is what is wanted.  I believe that the primary concern
here is actual files (which contain the sensitive content), but I'll defer to
others on the list with more experience and expertise to comment further on
the impact of label changes to namespace components.

Thanks,
--David

> From: Myklebust, Trond [mailto:[email-masked]]
> Sent: Wednesday, September 19, 2012 1:00 PM
> To: Haynes, Tom
> Cc: Black, David; nfsv4 list

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 19 Sep 2012 19:48:47 +0000

> From: Black, David [mailto:[email-masked]]
> Sent: Wednesday, September 19, 2012 3:26 PM
> To: Myklebust, Trond
> Cc: nfsv4 list
> Subject: RE: [nfsv4] CB_ATTR_CHANGED for Labeled NFS
> 
> > When do we expect a client-enforced Labeled NFS implementation to
> > change on the server? That basically means that the label is being
> > changed by some process that is not running on the client.
> 
> too much access - the classification needs to be changed and all now-
> disallowed usage of the document needs to be terminated ASAP with
> prejudice.
>
> > scenario to avoid races: if the label changes on the server after
> > we've checked the MAC, but before we issue the create operation (or
> > whatever), then we can't undo that operation or cancel it.
> 
> may already have (or have permitted an application to have) access that
> should not have been granted in 20/20 hindsight.  Prompt action to prevent
> further MAC-disallowed access is what is wanted.  I believe that the primary
> concern here is actual files (which contain the sensitive content), but I'll defer
> to others on the list with more experience and expertise to comment further
> on the impact of label changes to namespace components.
> 

So the concern is dealing with revoking access to already-cached data? That does sound like a job for delegations.

> > No. The whole problem is that it won't be limited to open files.
> > Labels affect all operations on a file/directory/symlink/...
> > Everything from a lookup or a readlink to a rename or unlink operation
> > can be subject to a MAC.
> >
> > > Does the client have to detect the server reboot (which it will do
> > > anyway) and then inform the server about files it has open? Does it
> > > do this with another OPEN? Or does the fact that the client presents
> > > the FH and a matching subject label really represent what we are
> > > after? (Which I think it does for this discussion.)
> > >
> > > While much of this can be handled by a client actually
> > > probing/polling for the security attribute, there are scenarios
> > > above where the server may want to take no action for a small delta of
> time.
> >
> > When do we expect a client-enforced Labeled NFS implementation to
> > change on the server? That basically means that the label is being
> > changed by some process that is not running on the client.
> >
> > scenario to avoid races: if the label changes on the server after
> > we've checked the MAC, but before we issue the create operation (or
> > whatever), then we can't undo that operation or cancel it.
> >
> > > We've already tackled all of these issues in 3550 and 5661 for locks
> > > and delegations.
> > >
> > > Redoing it all for this one callback is Trond's assertion that both
> > > approaches are heavy.
> > >
> > I'm not so much worried about coding it as I am about the server
> > having to track huge quantities of files.
> >
> > --
> > Trond Myklebust