Subject: [nfsv4] Server Side Copy URLs - security issue

From: <[email-masked]>
Date: Thu, 19 May 2011 22:05:53 -0400

>From the agenda for today's call:

  + Heterogeneous SSC and filenames with AUTH_SYS

  David Black has pointed out with that the approach of using
  the FH in this path:

  nfs://192.168.33.18//_COPY/10.11.78.56/_FH/0x12345

  is too predictable and could be used to intercept the
  file. His proposal is to have a requirement that a randomized
  component be presented.

>From the minutes of the call:

  + Heterogeneous SSC and filenames with AUTH_SYS

  We could not see the problem here. If the security flavor is AUTH_SYS,
  then the implication is that the server trusts everything presented to it.
  If the server does not want to trust, then it must use RPCSEC_GSSv3.

  AI: Tom to follow up with David Black.

---------------------------------------

Ok ... this seems to have suffered from a "whisper down the lane" effect.
Here's the complete explanation in one place:

The attack is against the inter-server copy techniques described in Sections
4.4.1.3 and 4.4.1.4 of draft-ietf-nfsv4-minorversion2-02.txt.  These techniques
do not use RPCSEC_GSSv3 and, crucially, *do not require* use of authentication
between the NFSv4 servers (e.g., 4.4.1.3 allows AUTH_NONE).  The source NFSv4
server passes a URL or URLs to the destination server via the NFSv4 client that
is doing the copy; the destination server copies from the source server by
using that URL.  The attack proceeds as follows:

Mallory wants to steal a sensitive source code file.  Mallory knows that Alice
is going to copy that file from NFS server Source to NFS server Destination.
Both Alice and Mallory are users on both NFS servers (this is an inside attack),
but Mallory has no access to the file on either server (either where it is on 
the Source server or where Alice is going to copy it to on the Destination
server).

Mallory waits until Alice issues her COPY_NOTIFY to the Source NFS server.
Mallory can predict the URLs that the Source server will generate and send
to Alice, so he immediately issues a COPY command to the Destination NFS Server
using those URLs (and somehow arranges to block or delay Alice's COPY).  The
Source NFS server sees the Destination NFS server access the source code file
using the expected URL and hence cooperates with Mallory's theft of the file
(it gets copied to a directory that Mallory has access to on Destination).

Needless to say, this is a security problem ... the countermeasure is to use
unpredictable URLs (need to put serious randomness in them) and some version of
RPCSEC_GSS to encrypt Alice's traffic to/from the servers so that Mallory can't
learn those URLs (and the inter-server traffic may also need to be encrypted).

I suspect that this attack extends to use of AUTH_SYS between the servers by
having Mallory run his own NFS server, Rogue, which claims to be doing the copy
from Source on behalf of Alice.  That's considerably more difficult because
Rogue has to use the same IP addresses as Destination (Alice sent the netloc
for Destination in her COPY_NOTIFY - Source had better be checking that when
Destination makes contact).  Doing so is likely to attract attention from the
network admin.  If Mallory can pull off this considerably more difficult attack,
then the call's conclusion that the countermeasure is RPCSEC_GSSv3 is probably
correct, but the current draft text allows the much simpler attack described
above.

At the very least, the security exposure here has to be covered in the Security
Considerations section, which is currently empty ;-).

Thanks,
--David
----------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 20 May 2011 10:29:07 -0500

On May 19, 2011, at 9:05 PM, <[email-masked]> <[email-masked]> wrote:

> 4.4.1.3 and 4.4.1.4 of draft-ietf-nfsv4-minorversion2-02.txt.

The draft neither mandates the format of these URLs nor that they must
be used to open the file. Both of these are implementation choices.

An implementation might chose to create the following URLs for the example
shown in 4.4.1.3:

      nfs://10.11.78.18//10.11.78.56//.copy/deadbeef.txt
      nfs://192.168.33.18//10.11.78.56//.copy/hellonurse.txt

Those names can be randomly generated. It all depends on what the
source wants to do.

(Note: the source should in this scenario disallow a READDIR inside
'.copy' in an effort to keep machines from periodically scanning the directory
to pick off files.)

The point is that the source has a portion of the namespace that no export
checks are performed and it can map the given filename to the file to
be copied.

An example compound to open might be:

   COMPOUND  { PUTROOTFH, LOOKUP ".copy" ; OPEN "hellonurse.txt" ; GETFH }

But even then, the source might have an export on '/'.

It turns out that what we want out of the netloc is the source IP for a given
subnet. We could actually care less about the file path and name.

If we look at what is contained in the COPY procedure, we see:

   struct COPY4args {
           /* SAVED_FH: source file */
           /* CURRENT_FH: destination file or */
           /*             directory           */
           offset4         ca_src_offset;
           offset4         ca_dst_offset;
           length4         ca_count;
           uint32_t        ca_flags;
           component4      ca_destination;
           netloc4         ca_source_server<>;
   };

The destination could instead do a compound:

   COMPOUND  {  OPEN (CLAIM_FH, <SAVED_FH>) ; GETFH }

The source could indicate its preference for the destination to connect by the manner
it constructs the netloc:

1) Open by name
      nfs://10.11.78.18//10.11.78.56//.copy/deadbeef.txt
      nfs://192.168.33.18//10.11.78.56//.copy/hellonurse.txt

2) Open by filehandle
       10.11.78.18
       192.168.33.18

3) Destination picks:
      10.11.78.18
      nfs://10.11.78.18//10.11.78.56//.copy/deadbeef.txt
      nfs://192.168.33.18//10.11.78.56//.copy/hellonurse.txt
      192.168.33.18

The first requires the source to be able to determine that a SSC is taking place and
bypass export checks. The benefit is that the source can use unique naming to do
a basic check to make sure that the destination is access the file via the correct
subnet.

The second requires only that the source make an exception (if necessary) for
the destination to access the SAVED_FH.

>  These techniques

The counter argument is that in an AUTH_SYS environment, trust is implicitly
granted.

This appears to fail in that exports are used to control which machines can connect to
portions of a server's namespace.

If the client is doing the copy:

cp /t/source/foo/name.tar.bz2 /t/dest/bar

then the client MUST have read permission to the file and write permission in
the directory. Further, it must have been able to traverse the respective namespaces.

Once we go to SSC, the same requirements hold, but we cannot make any
assumptions that the destination either can traverse the source's namespace
or have read permission to the file.

With AUTH_SYS and RPCSEC_GSSv[1,2], we are implicitly granting the destination
to have read permission to the file on the source. With RPCSEC_GSSv3, we are not,
we still make an explicit check.

Section 4.4.1.3 presents an argument of how the source implicitly grants such permission,

> Destination makes contact).  

The source cannot be making that check. The COPY_NOTIFY takes as a parameter
a single netloc. This is the location the client would be using if it were doing the copy.
The COPY_NOTIFY returns an array of netloc, which are all of the interfaces it will
accept requests from the destination on. But at that point, the source has no idea
what the source IP will be.

Section 4.4.1.3 talks about this as part of the multi-homed hosts.

Also, if the destination is actually part of a cluster, it might do some load balancing
and use a different node to contact the source.

A source could try to manage all of this by only returning one netloc in the
COPY_NOTIFY reply and only accepting an OPEN from the one netloc
it was presented with in the argument to COPY_NOTIFY.

> Doing so is likely to attract attention from the

There is an entire section on SSC Security Considerations:

4.4.  Security Considerations

For now, I'd prefer to keep such sections local to the feature being
discussed. I have yet to decide whether in the final document I will
retain that or pull them all into one place.

I.e., the final section for Security Considerations might consist of:

For SSC, see Section 4.4.
For LNFS, see Section 8.4.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Fri, 20 May 2011 14:40:55 -0400

Tom,

Thanks for the response - I'm selectively quoting the portions that apply to my

> Those names can be randomly generated. It all depends on what the

That's the crux of my concern.  As I currently understand the situation, an
implementation that chooses to not generate the names with sufficient randomness
creates a security exposure.  Documenting that exposure in the draft should be

> '.copy' in an effort to keep machines from periodically scanning the directory

Beyond that ...

> Section 4.4.1.3 presents an argument of how the source implicitly grants such permission,

My concern is that "implicitly granting the destination to have read permission"
implicitly bypasses access controls in the source filesystem.  That's unexpected
behavior, to put it mildly, especially as a warning that access controls are being
bypassed is nowhere to be found in 4.4.1.3.  IMHO, that's a significant omission.

> what the source IP will be.

I stand corrected.  I think what's needed here is a discussion of destination server
authentication to the source server - that should block Mallory's Rogue server when
AUTH_SYS is used between the source and destination servers, because Rogue should be
unable to authenticate as a trusted server (in the sense that Source believes the
AUTH_SYS info that is sent).

A couple of well-placed MUST requirements could go a long way towards addressing this
entire situation:
- Source server MUST authenticate destination server.
- Destination server MUST pass user identity from COPY to the source server
	and the source server MUST NOT perform the copy if that identity has
	no access to the source file(s).
As a consequence AUTH_NONE MUST NOT be used between the Source and Destination servers.

Adding randomness to the URL is an alternative to the above requirements - if the
two "MUST" requirements above cause problems, a possible structure is to provide
two alternatives - if the source server does not meet the above requirements, then
it MUST use a minimum amount of unpredictable randomness in the generated URLs.

Of course, use of RPCSEC_GSSv3 satisfies all of the above, but these are sections of
the draft that talk about what to do when RPCSEC_GSSv3 is not used.

Thanks,
--David

> -----Original Message-----
> From: Thomas Haynes [mailto:[email-masked]]
> Sent: Friday, May 20, 2011 11:29 AM
> To: Black, David
> Cc: [email-masked]
> Subject: Re: [nfsv4] Server Side Copy URLs - security issue
> 
>

---

From: Thomas Haynes <[email-masked]>
Date: Mon, 23 May 2011 16:15:05 -0500

> the first step, and that documentation should include this advice:

I agree we can call out better that the URL format is an implementation choice
and the possible security concerns.

> bypassed is nowhere to be found in 4.4.1.3.  IMHO, that's a significant omission.

I should have used 'explicitly ...'.

That is what the COPY does. I think we can illustrate this with an example that
shows the client have READ access and the destination not having READ access.
And we can show both URLs and OPEN in action here.

> - Source server MUST authenticate destination server.

It can do so by only presenting the destination server with a single endpoint, which
has to match the presented in the COPY_NOTIFY.

> 	no access to the source file(s).

I do not think this is a workable requirement outside of RPCSEC_GSSv3 - I do think
it is a MUST withing RPCSEC_GSSv3.

With the addition of using numeric ids into RFC3530bis, the destination server might
only have a couple of defined users on it. If it gets a request from a client, what is it
supposed to do?

1) If matching identities, use it.

2) If no matching identities:
a) Copy identity from client
b) Use root
c) Use nobody
d) Use some special identity we agree on

While each have pros and cons, in the end, I think we want 2a.

Of course, what happens in RPCSEC_GSSv[1|2] and the source
requiring kerberized access?

> As a consequence AUTH_NONE MUST NOT be used between the Source and Destination servers.

What if the export has AUTH_NONE as the only security flavor?

> it MUST use a minimum amount of unpredictable randomness in the generated URLs.

I think the right answer is 'SHOULD' in each scenario. We only require a 'SHOULD' for
RCPSEC_GSS_v3, so I don't see a stronger requirement for a less secure scheme.

---

From: <[email-masked]>
Date: Wed, 25 May 2011 00:28:21 -0400

Tom,

> And we can show both URLs and OPEN in action here.

Then we're being sloppy about what "destination" means.  Back to the original example:
- Alice is clearly explicitly granting the Destination server permission to read
        the file on her behalf.
- Alice does not intend to grant the Destination server permission to read the
        file on anyone else's behalf, including Mallory.
Currently 4.4.1.3 does both, and hence enables Mallory's theft of the sensitive file.  I
believe that's a security problem that has to be fixed, not left to the discretion of
the implementer.

I've outlined two ways to correct this problem:
(1) Require unpredictable URLs and say that in order to prevent this
        attack, copy communications with the NFS servers have to be
        encrypted (note: this is not "MUST use" for encryption in general,
        it only mandates encryption use when this attack needs to be
        prevented).
(2) Pass enough information between the Destination and Source servers
        so that the Source server can distinguish Alice's COPY from Mallory's
        and reject the latter.

It looks like you're favoring (2).  Please write that up in the next version
of the draft, after which we can debate whether and under what conditions the
requirements are "SHOULD" vs. "MUST".

> RCPSEC_GSS_v3, so I don't see a stronger requirement for a less secure scheme.

The problem I have is that server side copy is delivering weaker security than
results from the client doing the copy itself (in this long-running example,
if Alice copies the file herself, Mallory can't intercept it).  That's not good.

Thanks,
--David

> Sent: Monday, May 23, 2011 5:15 PM
> To: Black, David
> Cc: [email-masked]

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 25 May 2011 01:06:33 -0500

>        file on anyone else's behalf, including Mallory.

And the only way to enforce this is to use RPCSEC_GSSv3. The notion of secure
credentials in AUTH_SYS does not exsit and in RPCSEC_GSSv{1|2} we do not
have a means to tunnel those credentials.

>        it only mandates encryption use when this attack needs to be

Changing the URL is not sufficient - the FH also goes across the wire. If
we are going to assume that Mallory can capture the COPY_NOTIFY operation
or the COPY, then we have to assume that he can get the FH and intercept.

>        so that the Source server can distinguish Alice's COPY from Mallory's

This is not possible with the current requirements for COPY.  I.e., I'm sure you
could arrange something where the source sent an encrypted challenge
phrase to the client, which then passed it to the destination. It in turn would
generate a reply that only it could do, and send it to the source.

But that sounds heavy and we already have a secure method to do all of
this - RPCSEC_GSSv3.

> requirements are "SHOULD" vs. "MUST".

I'm in favor of mentioning them both in the Security Considerations.

My issue with all of this is that with AUTH_SYS the most you can do is
either create a small window or force enough constraints to get as
much security as is already present.

I.e., to get to the same level of security as already present in AUTH_SYS,
we could make a requirement that the destination MUST have read permission
to the file. It MUST be in the export list.

At that point, the COPY does not require weaker security than AUTH_SYS.

What are the four security checks that can be done?

DAC
Export 
ACL
MAC

With AUTH_SYS, these can all be spoofed by someone who has root access
to a client. With RPCSEC_GSSv1 and RPCSEC_GSSv2, it is not possible
for the authority to be delegated.

With RPCSEC_GSSv3, we have a COPY that meets your requirements.

> if Alice copies the file herself, Mallory can't intercept it).  That's not good.

Actually, I disagree. If Mallory has enough resources to force his client to fudge
up a NFSv4.1 request to copy the file, then he has the ability to passively capture
packets as they go across the wire.

And your network admin will not be able to detect that occurring in any logs.

David, while I am not adverse to mentioning these points in the security considerations
section, I believe we should not be mandating them. We already have a solution
which addresses your points, it is RPCSEC_GSSv3.

---

From: <[email-masked]>
Date: Wed, 25 May 2011 02:29:32 -0400

> or the COPY, then we have to assume that he can get the FH and intercept.

It is not necessary to assume that Mallory can capture either operation -
here's what I originally wrote:

Mallory waits until Alice issues her COPY_NOTIFY to the Source NFS server.
Mallory can predict the URLs that the Source server will generate and send
to Alice, so he immediately issues a COPY command to the Destination NFS Server
using those URLs (and somehow arranges to block or delay Alice's COPY).

There are other places in which the original description of the attack has
been departed from, so instead of continuing this thread ...

> I'm in favor of mentioning them both in the Security Considerations.

... I'll take you up on that offer.  Write up the attack and whatever you want
to say about what countermeasures are vs. aren't justified in the next version

> At that point, the COPY does not require weaker security than AUTH_SYS.

I'll take a look at that text, and we can take it from there.

Thanks,
--David

> Sent: Wednesday, May 25, 2011 2:07 AM
> To: Black, David
> Cc: [email-masked]