Subject: [nfsv4] server-side copy offload I-D

From: James Lentini <[email-masked]>
Date: Tue, 31 Mar 2009 17:06:57 -0400 (EDT)

I've posted a new individual draft co-authored with several colleagues 
that describes a set of NFS server-side copy offload procedures. The 
draft is available at:

 https://datatracker.ietf.org/drafts/draft-lentini-nfsv4-server-side-copy/ 

The proposed operations allow an NFS client to offload a file copy to:

 - a single NFS server
 - between two NFS servers

We look forward to feedback on this proposal.

-james

--
James Lentini | NetApp | 781-768-5359 | [email-masked]

---

From: Tom Haynes <[email-masked]>
Date: Tue, 31 Mar 2009 17:32:50 -0500

3.3.  Operation U: COPY_REVOKE - Revoke a server's copy priveleges
                                                        ^^^ privileges by the Thunderbird spellchecker

   ... However, if the
   client wishes to write sensitive information to file X that NFS
   server B is not intended to see, the client MUST use a COPY_REVOKE
   operation to ensure NFS server B is prevented access to the updates
   to file X.

This implies to me that the COPY_REVOKE operation works per file and not
per server. As the operation is defined on a per server basis, it does
raise the issue of what happens when user A decides that write sensitive
information is in danger of being copied, but user B does not have that
constraint.

Does user A have the right to halt the copy for user B?

What if that is a 2G iso image that is 99% done? I.e., I've just lost
a 40 minute transfer.

Consider this, under the current approach, I use an application, say
'cp' to copy data from server A to server B. I have the expectation that
root might stop that copy, but I don't expect that my office mate has
that ability.

---------

3.4.  Operation V: COPY - Copy a file on the server

   The SAVED_FH must be a file.  If SAVED_FH is not a file, the
   operation MUST fail and return NFS4ERR_ISDIR.

What if the SAVED_FH refers to a fifo, socket, etc?

Do you want to say "regular file" here?

---------

4.  Security Considerations

   For copy operations between distinct source and destination NFS
   servers, the authors expect RPCSEC_GSSv3 to provide a mechanism for
   the client to delegate authority to the destination NFS server to
   pull the data from the source NFS server.

(It would be nice here to provide a reference to the RPCSEC_GSSv3
draft proposal.)
 
I don't see that RPCSEC_GSSv3 is a requirement for NFSv4.1. And
"expect" is no where near as stringent as "MUST".

As worded, I think that the new operation has to work in an AUTH_SYS
environment. And I don't see that what is provided meets the security
requirements we expect there.

1) Export access checking is bypassed 

With this:

clientC $ cp /net/serverA/export/home/tdh/foo.tar.bz2 /net/serverB/export/home/tdh/foo.tar.bz2

clientC has to have access rights on both machines. And further more, the
user account has to have permissions to write on serverB.

Now it can be the case that clientC does not have write permission on serverB,
but serverA does have write permissions. Is it then valid to let this
operation proceed?

What about this sequence?

clientC $ cp trojan /net/serverA/export/home/tdh/trojan
clientC $ cp /net/serverA/export/home/tdh/trojan /net/serverB/export/home/tdh/trojan

I don't see enough detail in the proposal to prevent that from happening.

And note, I don't see RPCSEC_GSSv3 solving this problem. Exports/shares are
per machine, not per user. 

2) Are user credentials passed on?

The owner of the file to be copied may not be the user making the copy. So what
credential applies here?

I don't see any wording that says the source server must store the credentials
supplied in the COPY operation and present them to the destination server.

If the credentials supplied by the client are not passed to the destination
server, then this fails:

uid of thomas is 1813
uid of tdh is 3530

[root@serverB ~]> chown thomas:staff /export/home/tdh
[root@serverB ~]> chmod 766 /export/home/tdh
[root@serverB ~]> rm /export/home/tdh/foo.tar.bz2
 
[root@serverA ~]> chown tdh:staff /export/home/tdh/foo.tar.bz2

[tdh@clientC ~]> cp /net/serverA/export/home/tdh/foo.tar.bz2 /net/serverB/export/home/tdh/foo.tar.bz2

With the current wording in the proposal, this will work. And it should fail.

Okay, after reading Nico's draft (http://tools.ietf.org/html/draft-williams-rpcsecgssv3-00)
I do see that this issue should be addressed by RPCSEC_GSSv3.

---

From: James Lentini <[email-masked]>
Date: Tue, 31 Mar 2009 18:50:56 -0400 (EDT)

Tom,

Replies below:

> 3.3.  Operation U: COPY_REVOKE - Revoke a server's copy priveleges

Oops. That slipped by. Thanks for pointing that out.
 
> information is in danger of being copied, but user B does not have that

There operation is intended to be per file and user. We'll clarify 
this in the next version.

> Does user A have the right to halt the copy for user B?

No.

> root might stop that copy, but I don't expect that my office mate has

The analogy would be that the file server administrator may have the 
ability to halt a copy depending on the file server's implementation.
 
> Do you want to say "regular file" here?

Yes, that is what we envisioned. 

> (It would be nice here to provide a reference to the RPCSEC_GSSv3

Good point. We'll add that in the next version.
 
> "expect" is no where near as stringent as "MUST".

I would propose considering these operations for an NFSv4 version 
after .1.

In any event, you are right. RPCSEC_GSSv3 is required to secure these 
operations.

> but serverA does have write permissions. Is it then valid to let this

No, this should not be allowed. We will clarify this in the text.
 
> I don't see enough detail in the proposal to prevent that from happening.

What user is running cp? tdh?
 
> Okay, after reading Nico's draft
> (http://tools.ietf.org/html/draft-williams-rpcsecgssv3-00)

Do you also believe #2 needs to be addressed in the AUTH_SYS case?

---

From: Tom Haynes <[email-masked]>
Date: Tue, 31 Mar 2009 18:06:11 -0500

> this in the next version.

I think I struggled with that in general while reading the draft. I kept 
on wanting the operations to
be per file and per user. Instead they seemed to be per machine. 
Including per user information
would address my last concern below. But I just realized I don't know if 
that is what you meant
here by per user.

> What user is running cp? tdh?

Sure....

> Do you also believe #2 needs to be addressed in the AUTH_SYS case?

If these operations are allowed in the AUTH_SYS case, then yes. I.e., if 
RPCSEC_GSSv3 is
not being used to do the user authentication, then you either need to 
have the COPY* operations
do this or some sideband protocol. Myself, I'd prefer not to have a 
sideband protocol.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 31 Mar 2009 20:01:55 -0400

> > 1) Export access checking is bypassed With this:
> > 
> > clientC has to have access rights on both machines. And further
more, 
> > the user account has to have permissions to write on serverB.
> > 
> > Now it can be the case that clientC does not have write permission
on 
> > serverB, but serverA does have write permissions. Is it then valid
to 
> > let this operation proceed?
>

Note that export access checking is not defined or required in RFC3530
or the v4.1 spec.  It is something users expect even though the spec
doesn't define it.

The inter-server protocol is not defined in this proposal.  I think you
need to somehow say that it is up to the designer of the implementation
to prevent its normal security mechanisms (such as export access
checking) from being defeated, without having to go into any detail
about what those non-spec mechanisms might be.

-----Original Message-----
From: Lentini, James 
Sent: Tuesday, March 31, 2009 6:51 PM
To: Tom Haynes
Cc: [email-masked]
Subject: Re: [nfsv4] server-side copy offload I-D

Tom,

Replies below:

 
> This implies to me that the COPY_REVOKE operation works per file and 
> not per server. As the operation is defined on a per server basis, it 
> does raise the issue of what happens when user A decides that write 
> sensitive information is in danger of being copied, but user B does 
> not have that constraint.

There operation is intended to be per file and user. We'll clarify this
in the next version.

No.

> 'cp' to copy data from server A to server B. I have the expectation 
> that root might stop that copy, but I don't expect that my office mate

> has that ability.

 
> "expect" is no where near as stringent as "MUST".

I would propose considering these operations for an NFSv4 version after
.1.

> 1) Export access checking is bypassed With this:
> 
> clientC has to have access rights on both machines. And further more, 
> the user account has to have permissions to write on serverB.
> 
> Now it can be the case that clientC does not have write permission on 
> serverB, but serverA does have write permissions. Is it then valid to 
> let this operation proceed?

 
> And note, I don't see RPCSEC_GSSv3 solving this problem. 
> Exports/shares are per machine, not per user.
> The owner of the file to be copied may not be the user making the 
> copy. So what credential applies here?
> 
> I don't see any wording that says the source server must store the 
> credentials supplied in the COPY operation and present them to the
destination server.
> 
> If the credentials supplied by the client are not passed to the 
> destination server, then this fails:
> 
> uid of thomas is 1813
> uid of tdh is 3530
> 
> [root@serverB ~]> chown thomas:staff /export/home/tdh [root@serverB 
> ~]> chmod 766 /export/home/tdh [root@serverB ~]> rm 
> /export/home/tdh/foo.tar.bz2
> 

Do you also believe #2 needs to be addressed in the AUTH_SYS case?
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Tom Haynes <[email-masked]>
Date: Tue, 31 Mar 2009 21:15:10 -0500

> or the v4.1 spec.  It is something users expect even though the spec

Agreed and I tried to present things in a way of what I expected to happen.

> about what those non-spec mechanisms might be.

I'm confused, is it being defined or not?

Section 3.3 seems to talk about it.

What I think you mean is that the framework is there, but not the details.

I guess I'd rather see either a proposal that only did intra-server 
copies or one that does detail how
"normal" security mechanisms are enforced. I.e., otherwise how are two 
different vendors going to
be able to interoperate?

Is it an open protocol or not?

And as an abstraction, export checking is a name space issue. There is 
part of the name space on
serverB that clientC can see and part it can not (because it does not 
have export access). If
serverA can see part of the name space that clientC can not, then 
serverA can not do a file copy
into that area.

It gets harder if clientC has read-only access to that name space. They 
both have the same view of
the name space now, but only one can write to part of it.

What is needed to do these non-spec security mechanisms?

1) client IP
2) client machine credentials (kerberos ticket)
3) user/group
4) user credentials (kerberos ticket)

Anything else?

What if we flip the scenario? The clientC is allowed access, but serverA 
is not. That certainly works
with this example:

[tdh@clientC ~]> cp /net/serverA/export/home/tdh/foo.tar.bz2 
/net/serverB/export/home/tdh

As far as serverB is concerned, it really has no communication with 
serverA at all in this
transaction.

Do we want serverB to allow access if serverA says that it is tunneling 
a request for clientC?
There is some reason why it isn't allowed access.

Does this fall back to the "traditional" approach of the clientC reading 
the packets and then
writing them back out?

I guess if you want to say that inter-server COPY works if both servers 
are the same
implementation is okay, but I would prefer that we define such that it 
works in the face
of diversity.

We don't have to explicitly mention how the server deals with non-spec 
mechanisms,
because that would force us to add them to the spec, right?

But if we identify those parts of the packet that the destination server 
would be looking
at from the client, then the source server can supply them in the COPY 
procedure and
then let the destination server make the decision.

if (source_server is trusted) {
    if (source_client is trusted) {
        allow_copy();
    }
}

---

From: "Mike Eisler" <[email-masked]>
Date: Tue, 31 Mar 2009 20:21:04 -0700 (PDT)

> I'm confused, is it being defined or not?

The combination of RPCSEC_GSSv3 and COPY_NOTICE tells the source
server authorize the destination server to make a copy on behalf of
the user. If RPCSEC_GSSV3 is not being used, then the source server will
check if the user and the user's source network address has read access
according to the export policies of the destination server.

> Section 3.3 seems to talk about it.

It doesn't describe an inter-server copy protocol.

> Is it an open protocol or not?

Since the destination server is provided a filehandle of the source
file of the source server, it follows that the only open protocol would
be NFSv4.x, for x >= 2, assuming x = 2 is the minor version of NFSv4
that introduces inter-server COPY.
NFSv4.x + RPCSEC_GSSv3 has sufficient security mechanisms to allow
the right things to happen.

> serverA can not do a file copy

Export checking isn't a name space issue on most NFS servers. NFS servers
do not generally mapt the incoming filehandle to a path name and compare
it to the export policy. Instead, the fsid in the filehandle is checked
against the export table.

> the name space now, but only one can write to part of it.

Read-only access from the client to the source server's source file is all
that is needed.

> /net/serverB/export/home/tdh

If "cp" were re-implemented to use a copy() system call, then we
have:

clientC: COPY_NOTICE -> serverA
clientC: COPY -> serverB

serverB, perhaps using NFSv4.x, sends a series of READ requests to
serverA.

> As far as serverB is concerned, it really has no communication with

Until COPY that is.

> Do we want serverB to allow access if serverA says that it is tunneling

serverB has to check, when it receives a COPY from clientC,
that clientC has write access to /net/serverB/export/home/tdh/foo.tar.bz2 .
If it is using NFSv4.x to copy from serverA, it only has to send an
OPEN request to serverA for read acess. ServerA accepts or rejects
the OPEN according to whether user tdh has read access to the file,
and if clientC has read access to the file.

> if (source_server is trusted) {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

COPY is always sent to the destination so this check is not applicable.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 00:17:50 -0500

Ahh, you woke me up with your reply....

Mike Eisler wrote:
> the user. 

So RPCSEC_GSSv3 is sufficient to bypass export checking? I understand 
how it will
conceptually pass along the user credentials, but export checking is 
machine based and not
user based.

Hold off replying to this until you read the next part....
> If RPCSEC_GSSV3 is not being used, then the source server will

Shouldn't that be write access? Or have you flipped your servers around?

And to clarify, the source server can only do the check by attempting a 
write as that user? And
if I'm the destination server, I'm not going to trust that the source 
server has done the
authentication correctly.

I.e., RPCSEC_GSSV3 allows us to not put user credentials in the COPY 
procedures. But
if it is not being used, how do the user credentials and the machine 
credentials for that matter
get communicated from the source to destination server?

Either you have to add them to the COPY procedures or can you point out 
what in the
current protocol allows you to do this?

Also, even if the destination server did grant export access at the time 
of the start of the
copy, the export may be changed at any point during the sequence of  
WRITEs from the
source to destination. Thus any WRITE could trigger another access 
check, so where is
the information coming from?

> serverB, perhaps using NFSv4.x, sends a series of READ requests to

If we modified COPY_NOTICE to be:

                   struct COPY_NOTIFY4args {
                           /* SAVED_FH: source file */
                           /* CURRENT_FH: destination directory */
                           netloc4         cna_destination_server;
                   };

then the destination server could do the export check and directory
access check right then - we wouldn't have to worry about the source
server having to know this information. Conceptually, we wouldn't care
about RPCSEC_GSSV3 either - the access check is a metadata operation and
we don't need the user credentials during the writes...

The destination server would have to maintain state about the client
IP in case either the export access cache was flushed or the export
was modified. In either case, the server would be required to redetermine
the access rights.

Also, the destination server would not care whether the source server
had access rights or not. It would have to bypass export access checks
for this combination of (server, FH).

Finally, CB_COPY would need to be modified such that if the client failed
the export check as detailed above, the server could inform the client
that it no longer had permission to write the file.

If the destination server reboots, then it will not understand the WRITEs
coming from the source server. I.e., the source server may not have export
access permissions. If it does have write permissions, everything will seem
okay. If they fail, then the source server can contact the client and inform
it that it needs to re-establish the COPY_NOTIFICATION.

In looking at the draft, I can't tell if the CB_COPY is maintained between
the client and destination server or the client and the source server.

Like I said, I really liked this example you gave. I think providing
diagrams like this in the draft would make it much easier to understand
the flows going on here.

=====================

BTW - another nit in the draft:

   If the client supports the COPY operation, the client is REQUIRED to
   support the CB_COPY operation.

   The CB_COPY operation may fail for the following reasons:

   NFS4ERR_NOSUPP:  The copy offload operation is not supported by the
      NFS client receiving this request.

If it is REQUIRED to support CB_COPY, how can NFS4ERR_NOSUPP be generated?

Does this mean that the destination server has munged the client address
in memory and is contacting a client which does not support COPY?