Subject: Re: [nfsv4] server-side copy offload I-D

From: "Mike Eisler" <[email-masked]>
Date: Wed, 1 Apr 2009 04:44:25 -0700 (PDT)

On Wed, April 1, 2009 2:13 am, anshul wrote:
> On Wed, 2009-04-01 at 00:17 -0500, Tom Haynes wrote:
>> Mike Eisler wrote:
>> > On Tue, March 31, 2009 7:15 pm, Tom Haynes wrote:
>> >> Noveck, Dave wrote:
>> >>> The inter-server protocol is not defined in this proposal.  I think
>> you
>> >>> need to somehow say that it is up to the designer of the
>> implementation
>> >>> to prevent its normal security mechanisms (such as export access
>> >>> checking) from being defeated, without having to go into any detail
>> >>> about what those non-spec mechanisms might be.
>> >> I'm confused, is it being defined or not?
>> > The combination of RPCSEC_GSSv3 and COPY_NOTICE tells the source
>> > server authorize the destination server to make a copy on behalf of
>> > the user.
>> So RPCSEC_GSSv3 is sufficient to bypass export checking? I understand
>> how it will
>> conceptually pass along the user credentials, but export checking is
>> machine based and not
>> user based.

As Dave noted, export checking is not part of the NFS protocol.
Whether your server has export checking or not, and whether it
your server is willing to let the destination server acting as an NFS
client to copy data from it when authorized via COPY_NOTICE is
a decision for you, the implementer to make. How that is done is an
implementation choice. As specified, COPY_NOTICE provides a limited
override of any export policies that are in place.

> The destination server is being authorized read access to the source
> server via COPY_NOTICE, which is machine based.

More precisely, RPCSEC_GSSv3 + COPY_NOTICE authorizes a principal on
the destination server to read the file from the source server. The
principal on the destination server is likely a machine-based principal,
such as nfs/<dest-server-name>@EXAMPLE.COM. WHich makes sense, because
the only principals on the NFS servers that NFS clients are aware of, is
ths GSS principal name of nfs@<hostname> per the NFS standards.

>> Hold off replying to this until you read the next part....
>> > If RPCSEC_GSSV3 is not being used, then the source server will
>> > check if the user and the user's source network address has read
>> access
>> > according to the export policies of the destination server.
>> >
>> >
>>
>> Shouldn't that be write access? Or have you flipped your servers around?
>
> Read access should be checked at the source server and write access at
> the destination server via its export policies.

Yes. From the I-D:

"If the request is for a server-to-server copy, the source-fh is a
   filehandle from the source server and the compound procedure is being
   executed on the destination server."

>
>>
>> And to clarify, the source server can only do the check by attempting a
>> write as that user?

No, a READ.

>> And
>> if I'm the destination server, I'm not going to trust that the source
>> server has done the
>> authentication correctly.

The destination server doesn't care if the source server is checking
authentication correctly. All it cares about are:

1. does the user have write access to the destination?
2. will the source server let me (the destination server) read the file?

If both are yes, the COPY will happen.

>>
>> I.e., RPCSEC_GSSV3 allows us to not put user credentials in the COPY
>> procedures. But

COPY is inside COMPOUND. COMPOUND is an RPC. RPCs always have credentials.

If the destination server requires RPCSEC_GSS for the target file/directory,
the client will use RPCSEC_GSS. SECINFO/SECINFO_NO_NAME will negotiate this.
The credentials of the user that wants the file copied will be in
the RPCSEC_GSS or AUTH_SYS creds.

>> if it is not being used, how do the user credentials and the machine
>> credentials for that matter
>> get communicated from the source to destination server?
>>
>> Either you have to add them to the COPY procedures or can you point out
>> what in the
>> current protocol allows you to do this?
>
> Since COPY is always sent to the destination, the RPC headers should be
> able to provide the user credentials to the destination server. Machine
> credentials are provided by the COPY operation's arguments to the
> destination server.

The first sentence summarizes what I wrote. I believe what Anshul means by
the second sentence is that because the source server is identified in
COPY's arguments, the destination server will use its own principal
credentials when attempting to copy from the source server.

>> Also, even if the destination server did grant export access at the time
>> of the start of the
>> copy, the export may be changed at any point during the sequence of
>> WRITEs from the
>> source to destination. Thus any WRITE could trigger another access
>> check, so where is
>> the information coming from?
>
> The destination READs from the source and not vice-versa.

Yes. At first we were going to propose it the other way, but decided
that it would be easier if a source only had to grant read access to
a third party than granting write access to a third party. For the latter,
the issue isn't just that granting write access is more privilege, but that
we would have to deal with stuff like space reservations.

>
> Skipping some of next few questions, because they seem to based on a
> different premise.
>
>>
>> *Follow-up*: I really like this example:
>>
>> > If "cp" were re-implemented to use a copy() system call, then we
>> > have:
>> >
>> > clientC: COPY_NOTICE -> serverA
>> > clientC: COPY -> serverB
>> >
>> >
>> > serverB, perhaps using NFSv4.x, sends a series of READ requests to
>> > serverA.
>>
>> If we modified COPY_NOTICE to be:
>>
>>                    struct COPY_NOTIFY4args {
>>                            /* SAVED_FH: source file */
>>                            /* CURRENT_FH: destination directory */
>>                            netloc4         cna_destination_server;
>>                    };
>>
>> then the destination server could do the export check and directory
>> access check right then - we wouldn't have to worry about the source
>> server having to know this information. Conceptually, we wouldn't care
>> about RPCSEC_GSSV3 either - the access check is a metadata operation and
>> we don't need the user credentials during the writes...

As Anshul says, you appear to think that COPY is directed at the source.

Whether COPY is directed at the source or destination, some kind of
the delegation of authority from the client to another NFS server
is needed. We think that RPCSEC_GSSv3 holds a lot of promise.
We could do it without RPCSEC_GSSv3, such as by passing a secret in
a blob in COPY_NOTIFY, but if RPCSEC_GSSv3 is going to come to pass,
we'd rather do it with RPCSEC_GSSv3.

>>
>> The destination server would have to maintain state about the client
>> IP in case either the export access cache was flushed or the export
>> was modified. In either case, the server would be required to
>> redetermine
>> the access rights.
>>
>> Also, the destination server would not care whether the source server
>> had access rights or not. It would have to bypass export access checks
>> for this combination of (server, FH).
>>
>> Finally, CB_COPY would need to be modified such that if the client
>> failed
>> the export check as detailed above, the server could inform the client
>> that it no longer had permission to write the file.
>>
>> If the destination server reboots, then it will not understand the
>> WRITEs
>> coming from the source server. I.e., the source server may not have
>> export
>> access permissions. If it does have write permissions, everything will
>> seem
>> okay. If they fail, then the source server can contact the client and
>> inform
>> it that it needs to re-establish the COPY_NOTIFICATION.

Yes, all you are doing to moving the problem.

>>
>> In looking at the draft, I can't tell if the CB_COPY is maintained
>> between
>> the client and destination server or the client and the source server.
>
> The COPY is always sent to the destination server, thus the CB_COPY is
> always from the destination server to client.
>
>>
>> Like I said, I really liked this example you gave. I think providing
>> diagrams like this in the draft would make it much easier to understand
>> the flows going on here.

Point taken. Thanks.

---

From: James Lentini <[email-masked]>
Date: Wed, 1 Apr 2009 09:50:57 -0400 (EDT)

On Wed, 1 Apr 2009, Tom Haynes wrote:

> BTW - another nit in the draft:
> 
>   If the client supports the COPY operation, the client is REQUIRED to
>   support the CB_COPY operation.
> 
>   The CB_COPY operation may fail for the following reasons:
> 
>   NFS4ERR_NOSUPP:  The copy offload operation is not supported by the
>      NFS client receiving this request.
> 
> If it is REQUIRED to support CB_COPY, how can NFS4ERR_NOSUPP be generated?
> 
> Does this mean that the destination server has munged the client address
> in memory and is contacting a client which does not support COPY? 

Yes, that would be the idea.

If the client uses COPY, the text states:

   If the client supports the COPY operation, the client is REQUIRED to
   support the CB_COPY operation.

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 11:37:48 -0500

That is what the draft states:

> 3.2.  Operation T: COPY_NOTIFY - Notify a server of a future copy
>
>    ARGUMENTS
>
>                            /* CURRENT_FH: source file */
>    RESULTS
>                    union COPY_NOTIFY4res switch (nfsstat4 cnr_status) {
>                    case NFS4_OK:
>                            nfstime4        cnr_lease_time;
>                    default:
>                            void;
>                    };
>
>    DESCRIPTION
>
>    This operation authorizes a remote server to copy data from the
>    recipient.  It instructs the recipient to expect a copy operation
>    from the server identified by cna_destination_server on the file
>    specified by CURRENT_FH.
>
>    The cna_destination_server MUST be specified using the netloc4
>    network location format.  The server is not required to resolve the
>
>
>
> Lentini, et al.          Expires October 2, 2009                [Page 4]
> 
> Internet-Draft            NFS Server-side Copy                March 2009
>
>
>    cna_destination_server address before completing this operation.
>
>    If this operation succeeds, the recipient will allow the
>    cna_destination_server to copy the specified file exactly once before
>    the copy lease time cnr_lease_time expires.

This text states that the server receiving the COPY_NOTIFY will allow 
the other server to
drive the copy. If the COPY_NOTIFY is going to the destination, as you 
claimed in your email,
then the implication is that the source will be driving the writing of 
data to the destination.

The argument is the destination server. Why would I send that to the 
destination server if the copy
was destination reading from the source? Wouldn't I send the source 
address to the destination?

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 13:15:34 -0500

anshul wrote:
> On Wed, 2009-04-01 at 11:37 -0500, Tom Haynes wrote:
>   
>> Mike Eisler wrote:
>>     
> I did not understand how the draft states that the COPY is directed at
> the destination? See my replies below.
>
>   
>>>    recipient.  

Which is the remote server being mentioned here?

And which is the recipient?

The recipient has to be the server getting the COPY_NOTIFY, yes?

So, the remote server is authorized to copy data from the recipient.

If the recipient is the destination server, then this reads as the 
remote server
is authorized to copy data from the destination server. Which makes no sense
at all - regardless of which server is the remote server.

>>> It instructs the recipient to expect a copy operation

Again, the recipient is to expect a copy operation from the server 
identified by
the packet. Is that the same as the remote server from earlier?

Also, the name of the field is 'cna_destination_server', which states to me
that the destination server is driving the copy, but then why would you 
send the
address of the destination server to the destination server?

Mike mentioned at one point that you guys considered having the source 
server
driving the copy. Could the text be conveying that earlier model here?

>> drive the copy. 
> Correct.
>> If the COPY_NOTIFY is going to the destination
>>     
>
> But this is not correct. COPY_NOTIFY goes to the source.
>
> Regards,
> Anshul
>
>

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 13:18:20 -0500

> As Anshul says, you appear to think that COPY is directed at the source.

Agreed and again, some pretty pictures would have gone a long way...

But now you've changed the semantics of copy such that the destination 
needs read access to the source.

While you can argue that the COPY operations introduce new semantics, 
I'll argue that we do
not want to surprise users. The net change is that they should see 
faster copies and they won't
see the read/writes in packet traces with the client.

I can do:

[tdh@client ~]> cp /net/source/foo.iso /net/destination

And if we break that down, it means that:

1) The client needs read access to the source
2) The client needs write access to the destination

And by access, we mean both export and directory permissions. Although
if we wanted to be strictly protocol compliant, we could simply mean does
the user have permissions.

I can see this simply as:

   Client      Source   Destination
     +           +          +
     |           |          |
 0   |>--getfh-->|          |
     |          /|          |
     |         / |          |
 1   |<-------+  |          |
     |           |          |
 2   |>-------------getfh-->|
     |           |         /|
     |           |        / |
     |           |       /  |
 3   |<-----------------+   |
     |           |          |
     |           |          |
 4   |>--read--->|          |
     |          /|          |
     |         / |          |
 5   |<-------+  |          |
     |           |          |
 6   |>-------------write-->|
     |           |         /|
     |           |        / |
     |           |       /  |
 7   |<-----------------+   |
     |           |          |
     |           |          |
 8   |>--read--->|          |
     |          /|          |
     |         / |          |
 9   |<-------+  |          |
     |           |          |
     |           |          |
10   |>-------------write-->|
     |           |         /|
     |           |        / |
     |           |       /  |
11   |<-----------------+   |
     |           |          |

The question then becomes can we get the new COPY semantics such that there
does not need to be any permissions needed between the two servers? I.e.,
ignore which is driving the transfer of data....

Anything else and you either force the copy to revert to the traditional 
model or
you force the server admins to make some choices that they may not want to
make.

If we look at a simplified export model, one that only an access check on
the first operation which causes an access cache miss, then in the above
diagram, all of the permission checking is done at events 0 and 2.

We can mimic that in the new COPY with something like this:

(The getfh might be unnecessary, but the thought here is that
the client could determine whether or not it has read permission
on the source before it informs the destination that a file is
about to be created.)

   Client      Source   Destination
     +           +          +
     |           |          |
 0   |>--getfh-->|          |
     |          /|          |
     |         / |          |
 1   |<-------+  |          |
     |           |          |
 2   |>-------COPY_NOTIFY-->|
     |           |         /|
     |           |        / |
     |           |       /  |
 3   |<-----------------+   |
     |           |          |
     |           |          |
 4   |>--COPY--->|          |
     |          /|          |
     |         / |          |
 5   |<-------+  |          |
     |           |          |
 6   |           |>-write-->|
     |           |         /|
     |           |        / |
     |           |       /  |
 7   |           |<-----+   |
     |           |          |
 8   |           |>-write-->|
     |           |         /|
     |           |        / |
     |           |       /  |
 9   |           |<-----+   |
     |           |          |
10   |           |>-write-->|
     |           |         /|
     |           |        / |
     |           |       /  |
11   |           |<-----+   |
     |           |          |

The writes will need RPCSEC_GSSv3 for security.

The complication is when we remove the simple export model. Then
any write in the above figure could cause an export access check.
Granted that this is very implementation dependent, but we do
know that it can happen on at least two current implementations.

This is different than a server reboot...

A solution here is that the client has to be informed it has to
resecure access rights.

I can now see why you want to push all of the access checking to one
of the servers.

Going back to that model, what happens if the source server
has an export change such that the destination no longer has
access but the client does?

Does the CB_COPY occur, stating in effect that 45% of the file was
copied and it is up to the client to either restart the copy the
traditional way or is that an implementation issue and not covered
by this draft?

Or is the assumption that the access permission once granted can not
be revoked?

You are right that I am shifting the problem around, but which view
imparts the least surprise to the end-user?

---

From: James Lentini <[email-masked]>
Date: Wed, 1 Apr 2009 14:20:18 -0400 (EDT)

On Wed, 1 Apr 2009, Tom Haynes wrote:

> This text states that the server receiving the COPY_NOTIFY will 
> allow the other server to drive the copy. If the COPY_NOTIFY is 
> going to the destination, as you claimed in your email, then the 
> implication is that the source will be driving the writing of data 
> to the destination.

To be clear, a server-to-server copy is driven by the destination. As 
Mike noted, a COPY operation is directed at the destination (not 
the source).

> destination server if the copy was destination reading from the 
> source? Wouldn't I send the source address to the destination?

COPY_NOTIFY is sent by the client to the source server. It contains 
the destination server's identity. The client may then send the COPY 
operation to the destination server.

---

From: James Lentini <[email-masked]>
Date: Wed, 1 Apr 2009 14:38:25 -0400 (EDT)

On Wed, 1 Apr 2009, Tom Haynes wrote:

> If the recipient is the destination server, then this reads as the remote
> server

The remote server is cna_destination_server. 
 
> Again, the recipient is to expect a copy operation from the server identified
> by

Yes.

> that the destination server is driving the copy, but then why would you send
> the

The COPY_NOTIFY is sent to the source server.

> Mike mentioned at one point that you guys considered having the source server

The text should not convey that. I'll make sure that this is clear in 
the next version.

> > > This text states that the server receiving the COPY_NOTIFY will allow the
> > > other server to
> > > drive the copy.     
> > 
> > Correct.
> > 
> >   
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 13:39:42 -0500

Lets kill this part of the thread. James reply clears things up and I 
want to continue
over there.

Tom Haynes wrote:
> that the destination server is driving the copy, but then why would 
> you send the
>>> allow the other server to
>>> drive the copy.     
>>
>> Correct.
>>
>>

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 1 Apr 2009 13:30:05 -0500

> is needed. We think that RPCSEC_GSSv3 holds a lot of promise.

Great!  How do we move forward on RPCSEC_GSSv3?

Incidentally, Love Hörnquist Åstrand pointed out to me a few weeks ago
that we also need to expand the RPCSEC_GSS message sequence numbers from
32-bit to 64-bit to avoid rollover -- I don't want to think of an ONC
RPC/RPCSEC_GSS equivalent of TCP's PAWS! :)

> we'd rather do it with RPCSEC_GSSv3.

I never intended for RPCSEC_GSSv3 to include any delegation features
beyond what the GSS-API provides (and which is generally unused in NFS
clients and servers).  I'll have to think about that.

Nico
--

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 13:50:57 -0500

> operation to the destination server.

Agreed, I made that mistake.

I'm going to state again that some simple timing diagrams up front will 
clarify matters.

A new section before the Operations which described the typical 
scenarios would be
helpful. The above paragraph would be a great addition.

---

From: James Lentini <[email-masked]>
Date: Wed, 1 Apr 2009 16:04:55 -0400 (EDT)

On Wed, 1 Apr 2009, Tom Haynes wrote:

> But now you've changed the semantics of copy such that the destination needs
> read access to the source.

True, but the reads are performed with explicit permission from the 
user. The behavior is very similar to the original situation: data is 
read from the source and written to the destination. The system 
initiating the reads has changed, but the user context in which they 
are performed is the same.

> While you can argue that the COPY operations introduce new semantics, I'll
> argue that we do
> not want to surprise users. The net change is that they should see faster
> copies and they won't
> Anything else and you either force the copy to revert to the traditional model
> or

This would fall into the realm of a server-to-server copy protocol 
failure. The draft doesn't mandate a server-to-server copy protocol, 
so many possibilities are available. The server-to-server protocol 
might leverage NFS as you suggest or use something else.

> traditional way or is that an implementation issue and not covered 

Correct. If the server-to-server protocol fails, the destination 
server will perform a CB_COPY and indicate the number of bytes in the 
file that were copied. The client may then restart the operation using 
a COPY command with the correct offsets.

---

From: Nicolas Williams <[email-masked]>
Date: Wed, 1 Apr 2009 16:55:38 -0500

> clients and servers).  I'll have to think about that.

Oh, I see.  You want the destination server to use RPCSEC_GSSv3 to just
impersonate the original client to the source server for copying the
file.

Incidentally, that begs the question: what's the server-to-server copy
protocol?  Clearly you're intending to use an ONC RPC protocol using
RPCSEC_GSSv3.  What protocol could one use for this?  Oh! Oh! I know!
NFSv4!  :)

I can't think of any reason that the server-to-server copy protocol
couldn't just be NFSv4 itself.  I also can't think of any reason not to
REQUIRE that use of NFSv4 for this be supported.

Here's another question: how is a client to know whether
server-to-server copy will perform better than client mediated copy?
(When the servers are not the same, of course.)

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 1 Apr 2009 16:41:07 -0700 (PDT)

> Great!  How do we move forward on RPCSEC_GSSv3?

We need editors. Dave Quigley said last week he'd co-edit.
It sounds like he'd like help.

> RPC/RPCSEC_GSS equivalent of TCP's PAWS! :)

This is RPCSEC_GSSv3, auth flavor 6. Not RPCSECGSSv3, auth flavor 7.
Limit the scope, and you might actually get something done. Create a
whole new flavor, and you'll get nothing done.

Right now RPCSEC_GSS requires the context handles to be deleted on
seq# wrap around. There's no good reason for that.

I don't know what TCP PWAS is and what the issue is, but I'm pretty
sure TCP's problems with 32 bit sequence numnbers (distinguishing
seq#s of old connections from new connections is a mess) don't apply to
RPCSEC_GSS.

> clients and servers).  I'll have to think about that.

Granting privileges versus delegation: semantics. I don't care about
semantics.

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 1 Apr 2009 16:53:30 -0700 (PDT)

> impersonate the original client to the source server for copying the

No, we want the client to tell the source server to grant the destination
server read access to particular file. There's enough in RPCSEC_GSSv3
that we should be able to that.

> RPCSEC_GSSv3.  What protocol could one use for this?  Oh! Oh! I know!

NFSv4.x would be a fine protocol for cross-vendor (especially if
the source NFS server is sold by a non-NetApp vendor and the
destination server is sold by NetApp :-). For a single vendor,
a proprietary protocol that has intimate knowledge of the source and
destination file system formats might be a better choice.

> REQUIRE that use of NFSv4 for this be supported.

In NFSv4.x we have fs_locations_info, NFS4ERR_MOVED, and we also
have FedFS. The question of server-to-server protocols comes up, but
it has never been required to be answered in order to make
progress. Magic occurs. Mandates are not necessary. Clients are
happy.

If someone wants to take on the issue of server-to-server copy, migration,
and replication protocols, go ahead.

> (When the servers are not the same, of course.)

Since the client does not know what the server to server network
connection is, it doesn't. OTOH, the servers know, and each knows about
its connection to the client. And since servers are often behind the
glass wall where the nicer networking gear is, the client might as well
ask the servers. The servers can always say no.

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 1 Apr 2009 17:09:58 -0700 (PDT)

> imparts the least surprise to the end-user?

What's the difference in surprise? I don't see one.

Either the destination server has to authorize the source to do a
WRITE, or the source has to authorize the destination to do a READ.

There are a lot of disadvantages to a WRITE model:

- authorizing WRITE grants more power to a potentially out of control
  server

- less control over ENOSPC. In a READ model, if the destination is full,
  it can stop READing while it deals with its problem. In a WRITE model,
  the destination has to flow control the source when it runs out of
  space.

- less control over flow control. Sort of like the above but also consider
  that when doing a WRITE, an out of control client can WRITE chunks larger
  that what the server has said it write according to attributes and
  session parameters. READ questions however are bound in size. The amount
  the client can ask for us not bounded, but the server can return short
  READs. You can argue "so, with the push model, the destination can return
  a short WRITE". In the immortal words of the great Rob Thurlow when he
  reviewed the NFSv3 spec's WRITE procedure and puked: "Short WRITEs suck".

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 21:13:06 -0500

>> RPCSEC_GSSv3.  What protocol could one use for this?  Oh! Oh! I know!

Nico asks the question which really bugs me with this proposal. Why go 
to all of the bother
of defining the mechanics of initiating a server-to-server copy and 
leave out the guts?

> destination file system formats might be a better choice.

Provide a generic framework and let the two servers negotiate which 
server-to-server
copy method that they are going to use.

     enum copytype4 {
             COPY_NFSV4         = 1,
             COPY_NDMP          = 2,
             COPY_ZFS_SEND      = 3,
             COPY_NA_SNAP       = 4
     };

And we then provide a centralized agency to register copy methods.

Tongue-in-cheek? Sure.

But unless we have some protocol defined for server-to-server copies, 
how is the
client going to know it can send such a request? Doesn't it have to 
either assume
that the servers are different or does it have to be able to query the 
servers somehow
to determine if they are running server-to-server compatible protocol 
software?

---------------------

BTW:

   NFS4ERR_NOSUPP:  The copy offload operation is not supported by the
      NFS server receiving this request.
   NFS4ERR_PARTNER_NOTSUPP:  The remote server does not support the
      server-to-server copy offload protocol.

Where do we have something like NFS4ERR_PARTNER_PROPRIETARY? I.e., how
does a server give back an error code to the client that states it does
support a remote server from vendor A but not vendor B?

----------------

Or is NetApp going to provide a na-cp command for every NFSv4.x client 
implementation?

The enum could be as simple as:

     enum copytype4 {
             COPY_NFSV4         = 1,
             COPY_PROPRIETARY   = 2
     };

If the answer is COPY_PROPRIETARY, then that server is the one responsible
for determining if the other server is compatible. And perhaps this
is really a bitmask...

> progress. Magic occurs. Mandates are not necessary. Clients are

Yeah, magic scares me.

Mandates are still not necessary, but a basic framework can be put into 
place to take out
the guesswork.

My concern here is that we are defining part of the framework, saying 
that heterogeneous
server-to-server copy is too hard, and are going to get burnt when a 
user does use try
to do a copy in such an environment.

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 22:19:28 -0500

> WRITE, or the source has to authorize the destination to do a READ.

No, I'm proposing that the destination authorizes the COPY based on the 
client and not the
server. Which is the model that the user expects. The surprise is that 
now the servers need
access to each other.

Whether the resulting data transfer is via a READ or a WRITE doesn't 
matter to
my proposal. I just presented it in terms of a WRITE.

This works just as well:

  Client           Source   Destination
    +                +          +
    |                |          |
1   |>-COPY_NOTIFY-->|          |   Client is authorized to read
    |               /|          |   from the source
    |              / |          |
    |             /  |          |
2   |<-----------+   |          |
    |                |          |
    |                |          |
3   |>--COPY------------------->|   Client is authorized to write
    |                |         /|   to the destination
    |                |        / |
4   |<-----------------------+  |
    |                |          |
5   |                |<--read--<|
    |                |\         |
    |                | \        |
    |                |  \       |
6   |                |   +----->|
    |                |          |
7   |                |<--read--<|
    |                |\         |
    |                | \        |
    |                |  \       |
8   |                |   +----->|
    |                |          |
9   |                |<--read--<|
    |                |\         |
    |                | \        |
    |                |  \       |
10  |                |   +----->|
    |                |          |

The disadvantage that I see with this approach is that the destination 
may not
have read permission on the source and we have to do some magic to get this
to work. :->

>   reviewed the NFSv3 spec's WRITE procedure and puked: "Short WRITEs suck".
More stuff that could use a section in the draft -- talking about the 
issues of a READ versus WRITE model.

A design choice is made, document why...

---

From: Tom Haynes <[email-masked]>
Date: Wed, 01 Apr 2009 23:11:49 -0500

> ask the servers. The servers can always say no.

And behind that glass wall, everyone is trusted and we only need 
AUTH_SYS. Might as well make
an export of
serverA-be:/ -sec=sys,rw,anon=0

Why do the servers know? And does one server know about the other 
server's connection to the
client? Or do they need to get together and decide which method is faster?

And what in the above scenario implies that the destination server will 
have read access to the source?
The glass wall may look like a trusted environment to the outsider, but 
to an insider it may be a mess
of departmental turf wars.

I can envision a scenario where the source server is Marketing and the 
destination server is Engineering.
Neither server has any access rights to the other, we don't want to 
allow a Marketing droid to see a
development schedule and promise a feature to a customer. We also don't 
want a dashing Engineer
to see a proposed feature and implement it without a proper set of 
requirements. But we do want
a sysadmin on another locked down machine to be able to copy things from 
one to another.

In that scenario, the proposed COPY mechanism will fail.

Exports may not be in the spec, but they sure do get used a lot in 
reality. I'm pretty sure I still know
the names and ip of the NetApp Corp IT secure servers.

---

From: anshul <[email-masked]>
Date: Thu, 02 Apr 2009 13:37:12 +0530

On Wed, 2009-04-01 at 23:11 -0500, Tom Haynes wrote:
> AUTH_SYS. 

I think the example you give below proves that everyone is not trusted.

> Why do the servers know? 

The Servers may be able to determine based on historical data with a
particular client.

> And does one server know about the other 
> client? 

The other server would know about its own connection to the client, and
the recipient server can communicate with the other server.

> Or do they need to get together and decide which method is faster?

Correct. All this seems to be upto the server-to-server copy
implementation.

> In that scenario, the proposed COPY mechanism will fail.

Why do you think it would fail? RPCSEC_GSSv3 should be able to provide
mechanisms that allow delegation of authority for inter-server copy
appropriately as per user-based principals. So as long as we configure
the setup appropriately, I think we should be able to handle the
scenario you talk about. 

Regards,
Anshul

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 09:19:04 -0500

> that we should be able to that.

There's not enough in RPCSEC_GSSv3 for that right now, and, more
importantly, even though we could add enough it strikes as the wrong
layer for that (anything like a "capability" belongs above RPCSEC_GSS).

Impersonation, OTOH, is easy enough to do and RPCSEC_GSSv3 covers that
already.

> and replication protocols, go ahead.

I do not agree that just because we've punted on the server-to-server
migration and replication protocols we can now punt on any other
server-to-server protocol needs that may come up.

I think the litmus test should be just how much work it would take to
specify such a thing.  In this case it'd be practically no work at all,
thus let's not make cross-vendor interop worse.

> ask the servers. The servers can always say no.

That answers that question, thanks.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 09:26:02 -0500

> It sounds like he'd like help.

More than that we need _review_.  So far there's been no comments.

> seq# wrap around. There's no good reason for that.

Yes there is: RPCSEC_GSS requires that the replay_det_req_flag and
sequence_req_flag MUST[*] be turned off, which means that protection
against replays comes from the RPCSEC_GSS sequence number, which means
that the RPCSEC_GSS sequence number must not rollover.

But I agree that it's sufficient for now that new context handles be
established on sequence number wrap-around.  So what I'll do instead is
add/reserve a field/flag in RPCSEC_GSSv3 to negotiate larger sequence
number sizes and we can cross that bridge later.

[*] RFC2203 does not use RFC2119 terminology, sadly.

> Granting privileges versus delegation: semantics. I don't care about

No, it's much more than semantics: it changes the protocol
significantly.  I don't think we should add any delegation features to
RPCSEC_GSSv3 beyond what the GSS-API provides.  In any case, I think
that impersonation is sufficient for the cross-server copy protocol,
thus I'm happy.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 09:33:50 -0500

On Wed, Apr 01, 2009 at 10:19:28PM -0500, Tom Haynes wrote:
> No, I'm proposing that the destination authorizes the COPY based on
> the client and not the server. Which is the model that the user
> expects. The surprise is that now the servers need access to each
> other.
> 
> matter to my proposal. I just presented it in terms of a WRITE.

I agree with Mike on this: it's much better to have the destination READ
than the source WRITE.  But that's not what your point is about, I know.

> [...]
> may not have read permission on the source and we have to do some
> magic to get this to work. :->

No, this is easy: allow all servers in a cluster to impersonate any user
for READing, and RPCSEC_GSSv3 provides the bits needed for
impersonation.

Now, IF that's not enough because people don't wish to trust their
clusters' servers that much, THEN I think we need to add something like
a READ capability:

C->SRC: COPY_NOTIFY(FID, name of DST)
SRC->C: READ capability for DST to read FID on SRC
C->DST: COPY_START(name of SRC, FID, READ capability)
DST->C: result
DST->SRC: COPY_READ(FID READ capability)
SRC->DST: data
...
DST->C: CB_COPY_DONE(FID, status)

> >There are a lot of disadvantages to a WRITE model:

Indeed.

> A design choice is made, document why...

Sure.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 09:42:27 -0500

> The Servers may be able to determine based on historical data with a

The client _can't_ possibly know which copy method will be faster:
server-to-server or traditional.  That's because the client can't
possibly know what the network between the servers looks like.

The server could figure out which network is best for the copy, but more
likely it will just be configured to either accept server-to-server copy
requests or to reject them.  I think such configuration is quite
reasonable.

> > And what in the above scenario implies that the destination server
> > will have read access to the source?  The glass wall may look like a
> > trusted environment to the outsider, but to an insider it may be a
> > mess of departmental turf wars.
> > 
> > I can envision a scenario where the source server is Marketing and
> > the destination server is Engineering.  Neither server has any
> > access rights to the other, we don't want to allow a Marketing droid
> > to see a development schedule and promise a feature to a customer.
> > We also don't want a dashing Engineer to see a proposed feature and
> > implement it without a proper set of requirements. But we do want a
> > sysadmin on another locked down machine to be able to copy things
> > from one to another.
> > 

See my other replies today.  I do not believe that RPCSEC_GSSv3 is at
the right layer to do this sort of delegation.  RPCSEC_GSSv3 can do
impersonation though.

Nico
--

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 2 Apr 2009 10:59:10 -0400

> Nico asks the question which really bugs me with this 
> proposal. Why go to all of the bother of defining the 
> mechanics of initiating a server-to-server copy and 
> leave out the guts? 

The guts you propose is NFSv4, and that has been defined.
So the guts are not missing, at least in the core.  If you 
think there is something (and I'll bet it will be small that
would make NFSv4.2 be more able to fill this role, then 
propose it.  For now, I'll assume that NFSv4.1 is adequate.

> Provide a generic framework and let the two servers 
> negotiate which server-to-server copy method that they 
> are going to use.
>
>     enum copytype4 {
>            COPY_NFSV4         = 1,
>            COPY_NDMP          = 2,
>            COPY_ZFS_SEND      = 3,
>            COPY_NA_SNAP       = 4
>    };

> Tongue-in-cheek? Sure.

Why is it tongue-in-cheek?  I presume because any actual 
negotiation would be much simpler.  You try you specialized
proprietary method and if that is not supported then you 
use NFSv4.

The point is that adding a negotiation method would provide 
very little benefit for a lot of effort.  Similarly for 
specifying the inter-server protocol.  We are aware that 
NFSv4 exists and can be used.  We are aware that other 
protocols exists and can be used.  The guts exist but why 
do we need to define exactly your intestinal configuration?

> But unless we have some protocol defined for server-to-server 
> copies, how is the client going to know it can send such a 
> request? 

It can always send such a request.  If the server can't handle 
it, based on some sort of differences (although both would be
expected to support NFSv4), then it says it can't do it and 
the client uses OPEN, READ and GETATTR.

> Doesn't it have to either assume that the servers are different 
> or does it have to be able to query the servers somehow to 
> determine if they are running server-to-server compatible 
> protocol software?

That's all stuff the servers know far better and the way to 
query whether the server can do the copy is to ask it to.

> Or is NetApp going to provide a na-cp command for every NFSv4.x 
> client implementation?

There would have to be a copy syscall and VFS op, but that's
the case independent of any specification of server-to-server
protocol.  The point is NFSv4.2 does the COPY operation in
response to that VFS op and I would guess that other fs's
would return and error, and no server-o-server protocol is
going to change that.

> If the answer is COPY_PROPRIETARY, then that server is the 
> one responsible for determining if the other server is compatible. 
> And perhaps this is really a bitmask...

The server is always the one responsible for determining if 
the servers are compatible.  That's part of performing the 
operation.

> Yeah, magic scares me.

I can't remember who said it but, "Any sufficiently advanced
technology is indistinguishable from magic".

> Mandates are still not necessary, but a basic framework can 
> be put into place to take out the guesswork.

The basic framework that proposed is that the server is responsible
for doing the copy or telling the client that he can't.  Part of 
that is determining how to do the copy.  

> My concern here is that we are defining part of the framework, 
> saying that heterogeneous server-to-server copy is too hard, 
> and are going to get burnt when a user does use try to do a 
> copy in such an environment. 

Actually, I think we are saying it is too easy.  Of course the 
the servers can use NFSv4 to copy a file.  Since you don't think
we should mandate that it use that method, what else is there to
do here? 

> Where do we have something like NFS4ERR_PARTNER_PROPRIETARY? I.e., 
> how does a server give back an error code to the client that states 
> it does support a remote server from vendor A but not vendor B?

It really doesn't matter.  Any error is going to cause the client to

>     server-to-server copy offload protocol.

I don't think this is good because of the phrase "the server-to-server
copy offload protocol."  There isn't one defined.

   NFS4ERR_PARTNER_NOTSUPP:  The remote server does not support any of
      server-to-server copy offload protocols used by NFS server
receiving
      the request.

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Wednesday, April 01, 2009 10:13 PM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] server-side copy offload I-D

Mike Eisler wrote:
>> Incidentally, that begs the question: what's the server-to-server
>> copy protocol?  Clearly you're intending to use an ONC RPC protocol 
>> using RPCSEC_GSSv3.  What protocol could one use for this?  Oh! Oh! I
know!

Nico asks the question which really bugs me with this proposal. Why go
to all of the bother of defining the mechanics of initiating a
server-to-server copy and leave out the guts?

> NFSv4.x would be a fine protocol for cross-vendor (especially if the
> source NFS server is sold by a non-NetApp vendor and the destination 
> server is sold by NetApp :-). For a single vendor, a proprietary 
> protocol that has intimate knowledge of the source and destination 
> file system formats might be a better choice.
>
>   

Provide a generic framework and let the two servers negotiate which
server-to-server copy method that they are going to use.

     enum copytype4 {
             COPY_NFSV4         = 1,
             COPY_NDMP          = 2,
             COPY_ZFS_SEND      = 3,
             COPY_NA_SNAP       = 4
     };

Tongue-in-cheek? Sure.

But unless we have some protocol defined for server-to-server copies,
how is the client going to know it can send such a request? Doesn't it
have to either assume that the servers are different or does it have to
be able to query the servers somehow to determine if they are running
server-to-server compatible protocol software?

---------------------

BTW:

----------------

     enum copytype4 {
             COPY_NFSV4         = 1,
             COPY_PROPRIETARY   = 2
     };

If the answer is COPY_PROPRIETARY, then that server is the one
responsible for determining if the other server is compatible. And
perhaps this is really a bitmask...

>> couldn't just be NFSv4 itself.  I also can't think of any reason not 
>> to REQUIRE that use of NFSv4 for this be supported.
>>     
>
> In NFSv4.x we have fs_locations_info, NFS4ERR_MOVED, and we also have
> FedFS. The question of server-to-server protocols comes up, but it has

> never been required to be answered in order to make progress. Magic 
> occurs. Mandates are not necessary. Clients are happy.
>
>   

Yeah, magic scares me.

Mandates are still not necessary, but a basic framework can be put into
place to take out the guesswork.

My concern here is that we are defining part of the framework, saying
that heterogeneous server-to-server copy is too hard, and are going to
get burnt when a user does use try to do a copy in such an environment.

---

From: anshul <[email-masked]>
Date: Wed, 01 Apr 2009 14:43:55 +0530

Hi Tom,

Replies are inline -

On Wed, 2009-04-01 at 00:17 -0500, Tom Haynes wrote:
> Ahh, you woke me up with your reply....
> 
> Mike Eisler wrote:
> >>> The inter-server protocol is not defined in this proposal.  I think you
> >>> need to somehow say that it is up to the designer of the implementation

> > check if the user and the user's source network address has read access

> was modified. In either case, the server would be required to redetermine
> the access rights.
> 
> Finally, CB_COPY would need to be modified such that if the client failed
> If the destination server reboots, then it will not understand the WRITEs
> coming from the source server. I.e., the source server may not have export
> access permissions. If it does have write permissions, everything will seem
> okay. If they fail, then the source server can contact the client and inform
> In looking at the draft, I can't tell if the CB_COPY is maintained between

> =====================

Yes this seems to be one case for which the above could be applicable.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Wed, 01 Apr 2009 23:34:24 +0530

Correct.

Regards,
Anshul

---

From: James Lentini <[email-masked]>
Date: Thu, 2 Apr 2009 12:36:02 -0400 (EDT)

On Thu, 2 Apr 2009, Noveck, Dave wrote:

>       server-to-server copy offload protocols used by NFS server receiving
>       the request.

I agree Dave. I'll reword the draft to not refer to "the" 
server-to-server copy offload protocol.

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 09:41:25 -0700 (PDT)

> More than that we need _review_.  So far there's been no comments.

I suggest making your I-D easier to find so that it shows up in
http://tools.ietf.org/wg/nfsv4/ . Personal IDs that have -nfsv4-
after the author's surname in the I-D name will automatically show up.

> that the RPCSEC_GSS sequence number must not rollover.

In think this through, because RPCSEC_GSS lets the client arbitrarily slide
the window forward, I stand corrected.

> number sizes and we can cross that bridge later.

If it can be done in v3, it can be done in v4 (whether it
should be done is a different question). Let's limit the scope
to today's problems.

> [*] RFC2203 does not use RFC2119 terminology, sadly.

<shrug> There are lots of things in RFC2203 that would not past
muster with IESG, IANA, and the RFC editor today. The IESG members of
the time accepted RFC2203, and I suspect they aren't sad either. I'm not sad
about it, and don't care if anyone is.

The new NFSv4 WG charter let's us work on RPCSEC_GSSv0-bis if we think
we need to.

> RPCSEC_GSSv3 beyond what the GSS-API provides.  In any case, I think

Just like we shouldn't have added channel binding beyond what GSS
provided. :-)

> that impersonation is sufficient for the cross-server copy protocol,

What COPY needs is for a client to grant a temporary right to the
destination to read a file. In a subsequent rev of the I-D we'll provide
some details how we think RPCSEC_GSSv3 and privileges could enable this.

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 09:48:21 -0700 (PDT)

>> WRITE, or the source has to authorize the destination to do a READ.

And the difference in surprise is?

> 1   |>-COPY_NOTIFY-->|          |   Client is authorized to read

James should include the above in the next iteration of the I-D because
that's what we've proposed.

[...]
> A design choice is made, document why...

Agreed.

---

From: anshul <[email-masked]>
Date: Thu, 02 Apr 2009 22:20:15 +0530

> layer for that (anything like a "capability" belongs above RPCSEC_GSS).

Granting a caller privilege to initiate an inter-server copy to a
particular destination server for a particular remote server is on the
lines of the concept of privileges as defined in 'Section 3.  Privileges
and identity representation for NFSv4'.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Thu, 02 Apr 2009 22:20:47 +0530

> possibly know what the network between the servers looks like.

Yeah, I never said that the client would know.

> 
> The server could figure out which network is best for the copy,

Right, that is what I said.

>  but more

I have replied on that thread.

Regards,
Anshul

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 10:04:01 -0700 (PDT)

> of defining the mechanics of initiating a server-to-server copy and

After looking at the thumb nail sketch Anshul wrote on an internal
NetApp web page about how RPCSEC_GSSv3 privileges could be used to
authorize the destination to copy from the server, I now agree that
some description of how NFSv4 could be the copy protocol is needed.

> And we then provide a centralized agency to register copy methods.

I don't think the client has any business suggesting the copy
protocol. The servers can figure it out.

> client going to know it can send such a request? Doesn't it have to

It sends COPY_NOTICE to the source. Then it sends COPY to the destination.
if both succeed, it knows. If COPY_NOTICE fails, it knows.

> to determine if they are running server-to-server compatible protocol

Why does it have to assume anything?

In the responses to COPY and COPY_NOTICE.

> Or is NetApp going to provide a na-cp command for every NFSv4.x client

IOW, if Linux, Solaris, AIX, HP-UX, etc. fail to support COPY?
I would hope that Linux, UNIX, and Windows add a copy() system call.

There are other types of NFS clients though, ones that operate outside
the confines of POSIX and Win APIs.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 12:02:29 -0500

> Yeah, I never said that the client would know.

I didn't mean to imply that you had.  Tom raised the point and I agreed
with you that it's the server that must decide which copy approach to
use -- I was only elaborating on this.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 12:00:32 -0500

> after the author's surname in the I-D name will automatically show up.

That requires amending the WG charter, I think.  At the very least it
requires WG chair approval, which I asked for and did not get.

> about it, and don't care if anyone is.

The note was in case anyone wished to quibble with my interpretation of
the "must" that I was referring to.

> The new NFSv4 WG charter let's us work on RPCSEC_GSSv0-bis if we think

OK, I'll ask the chairs again.

> Just like we shouldn't have added channel binding beyond what GSS

That comparison fails to convince me for a number of reasons, not the
least of which being that there's no abstraction violation in one case
while there may be one in the other.

> some details how we think RPCSEC_GSSv3 and privileges could enable this.

Please do.  Also, please consider these alternatives:

 - the client tells the source to allow the destination to read the
   given file (this one implies that the client can reliably name the
   destination to the source in a way that the source can match when the
   destination contacts the source)

 - the client asks the source for a read capability that it then gives
   to the destination

 - the destination impersonates the client user to the source (this one
   implies cross-server trust)

Nico
--

---

From: Tom Haynes <[email-masked]>
Date: Thu, 02 Apr 2009 12:28:08 -0500

> the setup appropriately, I think we should be able to handle the

Because the mechanisms for RPCSEC_GSSv3 aren't spelled out in the document.

My guess is that your intent is for RPCSEC_GSSv3 to replace the 
export/access checks. Since
the draft doesn't talk about exports, whether this is true or not does 
not come out.

---

From: Tom Haynes <[email-masked]>
Date: Thu, 02 Apr 2009 12:32:29 -0500

Nicolas Williams wrote:
>  In any case, I think

Is that impersonation of a user or of a machine?

Or both?

---

From: Tom Haynes <[email-masked]>
Date: Thu, 02 Apr 2009 12:40:44 -0500

> think there is something (and I'll bet it will be small that

So then why not define it in this draft?

(And I've read ahead in the future and I see Mike is agreeing to do that.)

> propose it.  For now, I'll assume that NFSv4.1 is adequate.

The claim in the draft is that NFSv4.2 is adequate and NFSv4.1 is not.  
I.e., RPCSEC_GSSv3 is needed:

   For copy operations between distinct source and destination NFS
   servers, the authors expect RPCSEC_GSSv3 to provide a mechanism for
   the client to delegate authority to the destination NFS server to
   pull the data from the source NFS server.

And I can see that in the case of user credentials.

I still don't see how that works with exports.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 12:37:32 -0500

> Because the mechanisms for RPCSEC_GSSv3 aren't spelled out in the document.

Indeed.  Having the destination use RPCSEC_GSSv3 to impersonate the user
to the source will work with the current RPCSEC_GSSv3 I-D.

Having the client use RPCSEC_GSSv3 to delegate permission for a
particular operation will not work with the RPCSEC_GSSv3 I-D.  And I
think it shouldn't either because that means making RPCSEC_GSSv3 aware
of application-level operations -- yes, we can make those opaque to
RPCSEC_GSSv3, but RPCSEC_GSSv3 would still be dealing with
application-layer objects/bits while negotiating an RPCSEC-layer
feature, and that seems wrong, particularly when I think the same effect
can be achieved without changing RPCSEC_GSSv3 (see other posts of mine
on this matter from today).

> export/access checks. Since the draft doesn't talk about exports,
> whether this is true or not does not come out.

No, it's not about exports, but about authorization and delegation of
authority.  (Note: that's very different from delegation of
credentials.)

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 11:05:11 -0700 (PDT)

On Thu, April 2, 2009 10:00 am, Nicolas Williams wrote:
> requires WG chair approval, which I asked for and did nut get.

Anything after draft-williams- is up to you. I've got several
I-Ds prefixed as draft-eisler-nfsv4 and they all show up there. So
do the ones written by James Lentini and Dave Quigley. Visit the site;
seeing is believing.

You don't need permission to do this. The I-D upload tool cheerfully
and efficiently uploads whatever not draft-ietf name you want.
The IETF tools folks make sure it shows up as a related I-D automatically.

> while there may be one in the other.

OK, we agree to disagree.

>  - the client asks the source for a read capability that it then gives

Right. It can use RPCSEC_GSSv3.

>    implies cross-server trust)

You expect an author of RFC2203 to entertain cross-server trust as
an option? It's April 2, not April 1. :-)

---

From: anshul <[email-masked]>
Date: Thu, 02 Apr 2009 23:45:35 +0530

> on this matter from today).

As Mike mentioned, we would be posting some details on how we think
privileges can achieve this in a subsequent revision.

Yes.

Regards,
Anshul

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 02 Apr 2009 14:54:29 -0400

> I would hope that Linux, UNIX, and Windows add a copy() system call.

I seem to remember that the Linux CIFS folks tried to use their
equivalent of server-to-server copy as part of the 'sendfile()' syscall.
Unfortunately, that syscall internal implementation has since been
replaced, 'cos the whole sendfile() concept had intellectual property
issues. :-(

Cheers
   Trond

---

From: Tom Haynes <[email-masked]>
Date: Thu, 02 Apr 2009 14:10:02 -0500

>> authority.  (Note: that's very different from delegation of

So whether or not I trust an user is different than whether or not I 
trust a machine. I think I
can see how this works under RPCSEC_GSSv3.

To clarify, is the intent that when this happens:

[tdh@client ~]> cp /net/source/foo.tar /net/destination/archives

The COPY works if:

1) The client has read permissions on the source and write permissions 
on the destination?

2) The client has write permissions on the destination and the 
destination has read permissions on the source?

3) The client has write permissions on the destination and the user has 
read permissions on the source?

-----

Also, is the proposal mandating that the COPY mechanism MUST use 
RPCSEC_GSSv3?

If not, then how does this work under earlier versions and any of the 
AUTH_* flavors?

------

I.e., I'd like to see what is being proposed about how user checks will 
work and how export
checks will work - both with and without RPCSEC_GSSv3.

------

During async copy, does the client need to hold a read delegation on the 
source? What happens if it is recalled?

Does the client need to hold a write delegation on the destination? What 
happens if it is recalled?

---

From: "Iyer, Rahul" <[email-masked]>
Date: Thu, 2 Apr 2009 13:18:39 -0700

Inline....

-----Original Message-----
From: Madan, Anshul 
Sent: Thursday, April 02, 2009 9:51 AM
To: Nicolas Williams
Cc: Tom Haynes; [email-masked]
Subject: Re: [nfsv4] server-side copy offload I-D

Rahul> Sometimes, the client might not really care as long as the
throughput is "acceptable". In cases where people are running
hypervisors with NFS for storage, creating snapshots or clones entails a
series of READs and WRITEs from the host. This is a lot of CPU usage,
CPU that could have been used to run more VMs/provide better QoS for the
VMs. Same for the network bandwidth savings. 

Rahul> Or the admin could. My understanding is that admins will either
allow SERVERCOPY or disallow it, and clients will always try it, using
the READ/WRITE method as a fallback. Continuing from the previous point,
the decision of whether the server will support SERVERCOPY would be made
based on the server to server network, available protocols etc. While a
zero-configuration is nicer, this amount of minimal configuration
doesn't seem bad at all. 

Thanks,
Rahul

>  but more
> likely it will just be configured to either accept server-to-server 
> copy requests or to reject them.  I think such configuration is quite 
> reasonable.
> 

> > > will have read access to the source?  The glass wall may look like

> > > a trusted environment to the outsider, but to an insider it may be

> > > a mess of departmental turf wars.
> > > 

> > > access rights to the other, we don't want to allow a Marketing 
> > > droid to see a development schedule and promise a feature to a
customer.
> > > We also don't want a dashing Engineer to see a proposed feature 
> > > and implement it without a proper set of requirements. But we do 
> > > want a sysadmin on another locked down machine to be able to copy 
> > > things from one to another.
> > > 
> > Why do you think it would fail? RPCSEC_GSSv3 should be able to 
> > provide mechanisms that allow delegation of authority for 
> > inter-server copy appropriately as per user-based principals. So as 
> > long as we configure the setup appropriately, I think we should be 
> > able to handle the scenario you talk about.
> 

Regards,
Anshul

---

From: "Iyer, Rahul" <[email-masked]>
Date: Thu, 2 Apr 2009 13:28:24 -0700

Inline... 

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Wednesday, April 01, 2009 7:13 PM
To: Mike Eisler
Cc: [email-masked]
Subject: Re: [nfsv4] server-side copy offload I-D

know!

Rahul> This sounds a lot like the approach taken for pNFS in NFSv4.1.
The metadata server-data server protocol is left unspecified. In fact,
that's arguably the most important of the 3 protocols in question (the
client-metadata server and client-data server protocols being the other
two). The metadata server-data server protocol is what really makes pNFS
work! The other two are just merely scaffolding to expose, in a
standardized fashion, the clustered nature of storage systems today. I
really don't see why SERVERCOPY isn't any different.

Thanks,
Rahul

     enum copytype4 {
             COPY_NFSV4         = 1,
             COPY_NDMP          = 2,
             COPY_ZFS_SEND      = 3,
             COPY_NA_SNAP       = 4
     };

Tongue-in-cheek? Sure.

---------------------

BTW:

----------------

     enum copytype4 {
             COPY_NFSV4         = 1,
             COPY_PROPRIETARY   = 2
     };

Yeah, magic scares me.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 15:59:32 -0500

> privileges can achieve this in a subsequent revision.

BTW, I think I was confused about how you'd be using RPCSEC_GSSv3 in the
server-side copy protocol -- I think I get it now (though more details
would help).

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 15:57:25 -0500

> do the ones written by James Lentini and Dave Quigley. Visit the site;

Oh, I thought you meant I should make this a WG work item.  At this
stage I think re-submitting the I-D with a different individual
submission name is not going to be useful.  You found the I-D.  Read it,
review it, comment :)

There's much in RPCSEC_GSSv3 that could be controversial for all I know,
so a review by those who might find it controversial would be
particularly useful.  A review by the author of RPCSEC_GSSv1 would be
particularly useful (hint hint :)

> Right. It can use RPCSEC_GSSv3.

I've looked at draft-lentini-nfsv4-server-side-copy-00 again.  I think
what you mean is that the client tells the source what the destination's
name is, and then the destination has to authenticate as that same name.
Is that right?

That would be the first option above.

Why do you need RPCSEC_GSSv3 for that?  RPCSEC_GSSv1 will do for that.

Also, why put the destination server's address in the netloc4 type?  And
what if the server has multiple addresses and the source will see a
different one than the client?  Actually, the same could be true of the
server's authenticated name (though this is much less likely).  You may
want to add an operation by which to ask the destination what name and
address(es) it will use to read from the source.

Nico
--

---

From: James Lentini <[email-masked]>
Date: Thu, 2 Apr 2009 17:51:43 -0400 (EDT)

> Why do you need RPCSEC_GSSv3 for that?  RPCSEC_GSSv1 will do for that.

We need RPCSEC_GSSv3 because the destination not only has to identify 
itself, but also prove that the client authorized the destination to 
read the file.

> Also, why put the destination server's address in the netloc4 type?  

This allows for the greater flexibility in destination's identity. The 
server-to-server copy protocol may run over standard Ethernet or over 
something else (Fibre Channel, an RDMA fabric,...).

> And what if the server has multiple addresses and the source will 
> see a different one than the client?  Actually, the same could be 
> true of the server's authenticated name (though this is much less 
> likely).  You may want to add an operation by which to ask the 
> destination what name and address(es) it will use to read from the 
> source.

Wouldn't that be a configuration error?

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 15:18:45 -0700 (PDT)

> 1) The client has read permissions on the source and write permissions

yes

> destination has read permissions on the source?

yes

> read permissions on the source?

yes

and:

  4) The user has write permissions on the destination.

> Also, is the proposal mandating that the COPY mechanism MUST use

"MUST implement" not "MUST use". And COPY and COPY_NOTICE can
return NFS4ERR_WRONGSEC.

> If not, then how does this work under earlier versions and any of the

Since RPCSEC_GSSv3 is the only flavor that a user can use to authorize
another principal to do certain operations, pre-RPCSEC_GSSv3 flavors would
have go back to a trust model for the server-to-server copy, but even
then only if the user on the client has sent a COPY_NOTICE.

If the source and destination servers are nodes of a storage cluster,
then if the client is using RPCSEC_GSS (perhaps enforced by the servers'
use of _WRONGSEC), the model is very secure.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 17:03:12 -0500

> itself, but also prove that the client authorized the destination to 

Why?  Isn't it sufficient that the client tell the source to expect the
named destination to come knocking?  I should think so.

Sure, that's stateful, but the source could remember that only for a
short while and if the destination is late to get started, no problem ->
the copy will fail, the destination will callback the client, and the
client will restart or fallback on client-driven copy.

> something else (Fibre Channel, an RDMA fabric,...).

I think that's unnecessary.

> Wouldn't that be a configuration error?

Why would it be a configuration error to have a private network between
the servers?

Nico
--

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 15:41:18 -0700 (PDT)

> named destination to come knocking?  I should think so.

Not necessarily.

- The inter-server transfer protocol might not be as secure as RPCSEC_GSS.
It might be as weak as AUTH_SYS. E.g. it might be going over raw FC.

- As you've pointed out the name/address of destination outside the glass
might not be the same as it is inside the glass wall. Authenticating
nfs@<destination server> might not work.

> I think that's unnecessary.

You can do better than that. :-)

Today in NFSv4.1 fs_locations_info lets one specifiy an IPv4
address, an IPv6 address, or a FQDN. It struck me that might not
suffice behind the glass wall, where the IP protocol is sometimes
viewed with fear and loathing.

> Why would it be a configuration error to have a private network between

Good point.

It seems to me that if the destination address is different behind
glass, so in the source address. COPY_NOTICE could return the source
address that should be used. The COPY operation would then include
that source address. By virtue of RPCSEC_GSSv3 privilege,
the destination will have proven to the source that it is the server
authorized to read on behalf of the user.

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 02 Apr 2009 18:54:57 -0400

> itself, but also prove that the client authorized the destination to 

Why couldn't the source server encode that authorisation in the
filehandle that the destination server presents in the RPC call? Isn't
that the recommended approach made in section 13.1 to allow pNFS servers
to encode similar information?

Cheers
  Trond

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 2 Apr 2009 16:50:29 -0700 (PDT)

> filehandle that the destination server presents in the RPC call? Isn't

It could, but then anyone could use that filehandle to read the file.

> to encode similar information?

You mean this snippet from 13.1 of NFSv4.1 draft 29?

   If metadata server routing and/or identity information is encoded in
   data server filehandles, when the metadata server identity or
   location changes, the data server filehandles it gave out will become
   invalid (stale), and so the metadata server MUST first recall the
   layouts.  Invalidating a data server filehandle does not render the
   NFS client's data cache invalid.  The client's cache should map a
   data server filehandle to a metadata server filehandle, and a
   metadata server filehandle to cached data.

The data server filehandles don't grant authorization, they are used
to route READ and WRITE requests to the correct stripe unit and
to gracefully fence clients after layout revocation.

Besides which COPY_NOTICE doesn't change the current filehandle.

Now, granted, it could, and we could mandate that COPY_NOTICE be called
with RPCSEC_GSS privacy so that the filehandle is protected from attackers,
etc.

There are several, likely dozens or more, ways to do this. We've proposed
one. Before we noticed RPCSEC_GSSv3 we had the notion passing
shared secrets within COPY_NOTICE and COPY, but figured it made more
sense to use RPCSEC_GSS to do that sort of thing.

Just as there are dozens or more ways of doing COPY.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 23:47:49 -0500

> It might be as weak as AUTH_SYS. E.g. it might be going over raw FC.

Then using telling the source server what the destination server's IP
addresses are should suffice (or else you ought not be using AUTH_SYS).

> nfs@<destination server> might not work.

That's why you should have the client ask the destination what name it
will use when authenticating to the source server (and what IP addresses
too).

> You can do better than that. :-)

I didn't mean that FC is unnecessary -- only that what you want
RPCSEC_GSSv3 to do is unnecessary.

Sure.

> authorized to read on behalf of the user.

Please be careful with terminology.  If you mean "capability" then say
that, not privilege.  (What Linux calls capabilities used to be called
privileges in an earlier version of the POSIC draft from which they are
derived, and were always called privileges, while what everyone else
calls capabilities in the context of distributed filesystems has never
been called capabilities that I know.)

In any case, I think it's utterly inappropriate for RPCSEC_GSSv3 to
negotiate capabilities (in the sense of cryptographic tokens authorizing
a party to a particular operation on a particular object).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 23:51:02 -0500

> It could, but then anyone could use that filehandle to read the file.

You could always encode the destination server's name into that FH.

BTW, we call this sort of thing a "capability."

> to gracefully fence clients after layout revocation.

FHs are opaque, so they can be actual "capabilities" (but without a
protocol to handle re-keying you're left with volatile FHs as the only
way to make that work).

> with RPCSEC_GSS privacy so that the filehandle is protected from attackers,

Or you could encode the name of the principal that can use that FH in
the FH (see above).

You could also create a first-class capability in NFSv4 instead of
trying to contort RPCSEC to do it for you.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 23:54:16 -0500

> then only if the user on the client has sent a COPY_NOTICE.

Again, RPCSEC_GSSv3 as it currently stands provides no such feature.
What you seem to think it provides are capabilities (cryptographic
tokens) representing the authorization of one principal to act on an
object in some specific manner.  RPCSEC_GSSv3 provides no such thing.
Making it do so means making it aware of application layer operations
and objects, and though that could be and would have to be done in a way
that's opaque to RPCSEC_GSSv3, it's still an enormous change to the RPC
security model and I think we'll need to be sure we have no suitable
alternatives (I provided some in earlier e-mails, and just now in
another) first.

Nico
--

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 00:46:28 -0500

> Just as there are dozens or more ways of doing COPY.

So why not present  the other methods here and why you shot them down?

We keep on re-inventing the wheel on these design points...

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 01:04:17 -0500

> authorized to read on behalf of the user.

The client presents in the COPY_NOTICE 10.10.20.42 as the address of the 
destination server.
Or perhaps it presents foo.corp.company.com.

How does the source server know what the backend name/IP is for the 
destination server?

The COPY_NOTICE has to return a list of allowed source addresses. And 
then the destination
server has to decide which one to use.

An implementation might provide some means to allow the admin to 
configure the preferred networks
for server-to-server copies...

But in the context of the proposal, any valid path selected by the 
destination should be accepted
by the source via the virtue of RPCSEC_GSSv3.

---

From: anshul <[email-masked]>
Date: Fri, 03 Apr 2009 12:08:13 +0530

> addresses are should suffice (or else you ought not be using AUTH_SYS).

Not while using AUTH_SYS, but when we use RPCSEC_GSS and require more
security - there we would also like the destination to authenticate
itself.

> a party to a particular operation on a particular object).

I think the current Draft already grants 'RPCSEC_GSSv3 privileges' ->
'capabilities' on a per-suboperation basis - 

3.  Privileges and identity representation for NFSv4

   The representation of users and groups for use in identity assertions
   in RPCSEC_GSSv3 SHALL be the same as the user and group
   representations used by NFSv4 for access control list subjects on the
   wire, cast as an octet string ("opaque").

   The following privileges are defined for use with the NFSv4 protocol:

   file_chown  Generally allows the caller to change a file's owner
      regardless of who owns the file.

   file_chown_self  Generally allows the caller to change the owner of a
      file it owns.

   file_dac_execute  Generally allows the caller to read any file for
      execution.

   file_dac_read  Generally allows the caller to read any file or
      directory.

   file_dac_search  Generally allows the caller to search any directory.

   file_dac_write  Generally allows the caller to write to any file (or
      create/delete/link objects in directories).

   file_link_any  Generally allows the caller to create hardlinks to
      files not owned by the caller.

   file_owner  Generally allows the caller to modify the access,
      modification and other timestamps of a filesystem object, as well
      as its permissions and ACL.

   file_setid  Generally allows the caller to set the set-user-ID and
      set-group-ID bits of a file.

   file_downgrade_sl  Generally allows the caller to downgrade the
      security label of a filesystem object.

   file_update_sl  Generally allows the caller to upgrade the security
      label of a filesystem object.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Fri, 03 Apr 2009 12:10:23 +0530

On Thu, 2009-04-02 at 23:54 -0500, Nicolas Williams wrote:
> object in some specific manner.  

To clarify, not a capability to act on a specific application object but
the capability is generic. See my reply on the other thread,
'RPCSEC_GSSv3 privileges' already seems to provide generic capabilities
representing authorization to perform suboperations.

> RPCSEC_GSSv3 provides no such thing.
> and objects, 

Again, see my reply on the other thread.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Fri, 03 Apr 2009 12:27:58 +0530

> How does the source server know what the backend name/IP is for the 

Why does it need to? The server-to-server protocol can present
10.10.20.42 to the source server even if the destination server
communicates with the source server from a different IP say 1.2.3.4.

> by the source via the virtue of RPCSEC_GSSv3.

Even when using AUTH_SYS, any valid path selected by the destination
should be accepted by the source. RPCSEC_GSSv3 privilege allows the
server to prove that it is the authorized to read from the source.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Fri, 03 Apr 2009 12:43:51 +0530

> trying to contort RPCSEC to do it for you.

We are not contorting RPCSEC to do any thing involving application
objects.

Regards,
Anshul

---

From: James Lentini <[email-masked]>
Date: Fri, 3 Apr 2009 11:28:51 -0400 (EDT)

On Thu, 2 Apr 2009, Trond Myklebust wrote:

> equivalent of server-to-server copy as part of the 'sendfile()' 
> syscall. Unfortunately, that syscall internal implementation has 
> since been replaced, 'cos the whole sendfile() concept had 
> intellectual property issues. :-(

I hope that OSes will, eventually, be able to support a copy offload 
API in one form or another. As you mention, Windows already does with 
CIFS. At one point, you mentioned Linux's splice(2) as a potential API 
for supporting COPY. Do you still see that as a viable option?

Beyond OSes, there are NFS clients that are not below a traditional 
VFS layer and are unconstrained with these user/kernel API issues. 
These implementations will be able to take advantage of this feature.

While keeping implementation concerns in mind, we should focus on 
defining the protocol aspects of copy offload in this forum. If there 
is some aspect of the protocol that makes the implementation 
unreasonably difficult, the protocol can be modified, but I don't know 
of such an issue at the moment.

-james

---

From: James Lentini <[email-masked]>
Date: Fri, 3 Apr 2009 11:34:36 -0400 (EDT)

On Fri, 3 Apr 2009, Tom Haynes wrote:

> The COPY_NOTICE has to return a list of allowed source addresses. 
> And then the destination server has to decide which one to use.

This is what Mike is suggesting above. This strikes me as a good 
addition for the next revision.

BTW, the draft terms the operation COPY_NOTIFY, not COPY_NOTICE. I 
think we are all talking about the same operation though.

---

From: Trond Myklebust <[email-masked]>
Date: Fri, 03 Apr 2009 12:53:08 -0400

> for supporting COPY. Do you still see that as a viable option?

splice() is supposed to copy data between a file descriptor and a pipe,
doing zero-copy tricks to actually move the data around the kernel. You
can use it to do fast copies, by using 2 splice() syscalls (one to
zero-copy data from the source file into the pipe, and then one more to
zero-copy it back out into the destination file).

As you can see, though, the whole reliance on a pipe as an intermediate
object makes it a different kind of abstraction,  so I'm afraid I don't
think it will help.

Cheers
  Trond

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 12:03:04 -0500

> addition for the next revision.

Mike is suggesting using 1 address, I'm suggesting using all.

> think we are all talking about the same operation though.

Yes, my bad...

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 3 Apr 2009 12:15:08 -0500

> communicates with the source server from a different IP say 1.2.3.4.

Because the client should tell the source what principal name and IP
address(es) the destination will use when talking to the source -- this
way even RPCSEC_GSSv1 and AUTH_SYS will work in the destination->source
protocol.

> server to prove that it is the authorized to read from the source.

No.  Not at all.  You're missing the point about what RPCSEC_GSSv3
privileges are.

RPSEC_GSSv3 allows the client to assert privileges in the Solaris sense
of the word "privilege" and the server then decides what privileges to
actually grant the client based on: a) the client host principal's max
allowed privileges, b) the client user principal's max allowed
privileges.

The key point: the client's assertion of privileges proves exactly
_nothing_.

You and Mike seem to think that RPSEC_GSSv3's privileges are something
like cryptographic capabilities.  They are not.

Nico
--

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 12:24:51 -0500

> communicates with the source server from a different IP say 1.2.3.4.

But how does the destination server know to communicate with the source 
server at 10.100.10.42?

Look at the following:

               +------- 10.100.10/24 -------------+
               |                                  |
               | 10.100.10.43                     | 10.100.10.42
          +----+---+                       +------+------+
          | Source |                       | Destination |
          +----+---+                       +------+------+
               | 10.10.20.43                      | 10.10.20.42
               |                                  |
               |                                  |
-- Glass Wall  | -------------------------------- | ----------------
               |                                  |
               |                                  |
               +------- 10.10.20/24 -+------------+
                                     |
                                     |
                                     |
                                     | 10.10.20.192
                                +----+-----+
                                | Client   |
                                +----------+

Assume 10.100.10/24 is a 10G network and 10.10.20/24 is a 1G network.

The client sends a COPY_NOTIFY to the source. It sends it to 
10.10.20.43, because
it has no idea of 10.100.10.43 and can't even talk to that address. It 
states to expect
a COPY request from 10.10.20.42.

It then sends a COPY to the destination at 10.10.20.42. It states that 
the copy is
to occur with 10.10.20.43.

How does the destination know that 10.10.20.43 corresponds to 10.100.10.43?
And while I've lined the numbers up, you can't count on that always 
happening. And
in any event, how does the kernel know this?

The proposed change here is that the COPY_NOTIFY returns to the client 
the list
of:

10.10.20.43
10.100.10.43

And the client in turn sends that on in the COPY.

Now the destination sees that it can talk to source at either of the two 
addresses. And
seeing as how it has interfaces on both of those, it can chose either. 
We would just
prefer that it uses 10.100.10/24.

For some reason, I'm thinking an implementation would want some form of 
a simple
policy engine here to make that type of decision. :->

Really, the load might change during the time of day, we might want to 
detect that one
of the networks is down (or an interface to that network is down), etc.

But in the terms of the protocol, we don't care which of the offered up 
addresses is used.

> server to prove that it is the authorized to read from the source.

But even if valid, with AUTH_SYS, we don't know that it is authorized.

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 00:00:57 +0530

> way even RPCSEC_GSSv1 and AUTH_SYS will work in the destination->source

Using RPCSEC_GSSv1 for the destination->server protocol seems to be a
separate discussion altogether. I do not see any details on how that
would work.

> like cryptographic capabilities.  They are not.

Not exactly. I would prefer if you wait till we post details in a latter
revision before critiquing it.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 00:03:12 +0530

> But how does the destination server know to communicate with the source 

It does not, it can communicate from any IP. But since the client
request came to the destination IP at 10.100.10.42, the
destination-server protocol can present 10.100.10.42 to the source
server, even though it communicates from a different IP.

> Look at the following:

Yes I had understood the context before and my response is still the
same, the diagram and longer explanation was not needed :). Thanks
Anyway.

We know, see my reply above.

Regards,
Anshul

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 3 Apr 2009 13:38:32 -0500

On Apr 3, 2009, at 1:30 PM, anshul <[email-masked]> wrote:

>> way even RPCSEC_GSSv1 and AUTH_SYS will work in the destination- 
>> >source
>> protocol.
>
>

Please outline when that update is expected and the content that is  
expected to be new or updated.

Spencer

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 13:43:31 -0500

> Mike is suggesting using 1 address, I'm suggesting using all.

Also, notice that the COPY_NOTIFY will also have to present a list of 
possible addresses
that the destination server might present to the source.

This in turn suggests that the client will have to send something like 
COPY_PREP to the
destination server to get such a list of addresses.

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 13:47:39 -0500

> server, even though it communicates from a different IP.

No, you have missed the point!

Look at the diagram, the client request has to come to the destination 
IP at 10.10.20.42.

The client has no interfaces which connect to 10.100.10/24.

> same, the diagram and longer explanation was not needed :). Thanks

Yes, it is needed.

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 14:09:49 -0500

> server to prove that it is the authorized to read from the source.

So I think you can do the server-to-server copy without using 
RPCSEC_GSSv3 at all.

Look at the example:

[tdh@client ~]> cp /net/source/builds/osol_111.tar.bz2 
/net/destination/archives

What happens in the traditional approach:

1) The client tries to mount source:/builds. It needs to pass an export 
check on the source.
     a) It fails, copy aborted
 2) The client tries to start reading the file as the user 'tdh'. It 
needs to pass a permissions check.
     a) It fails, copy aborted
 3) The client to mount destination:/archives. It needs to pass an 
export check on the source.
     a) It fails, copy aborted
  4) The client tries to open (and/or create) the file as the user 
'tdh'. It needs to pass a permissions check.
     a) It fails, copy aborted
  5) The client cycles between reads and writes until the file is 
copied. Note that at any point an
       export can be modified on either server, resulting in an export 
check and an abortion of the
       copy if access is denied.

With just AUTH_SYS, we can model this with the COPY proposal:

1) The client needs to get a FH from the source. At this point, it will 
need to pass an export check on
    the source.
    a) It fails, copy aborted

2) The client sends a COPY_PREP to the destination.
    This will simply enumerate the possible addresses that the 
destination might use to connect any
    source server.

    Note that if this fails, it implies that the destination does not 
support server-to-server copies.

3) The client sends a COPY_NOTIFY to the source.At this point, it needs 
to pass a permissions check.
     a) It fails, copy aborted.
     b) It succeeds, it returns:

      list of addresses
      lease time
      stateid

      Note that the stateid is not encrypted. And also note that the 
source server should be allowed to have
      a call-back path to the client to revoke the copy.

At this point, the client has a tuple (source address, FH, stateid) that 
is unique on the source server. Furthermore,
only a host which comes from one of the addresses returned by the 
COPY_PREP _and_ presents this tuple to the
source server has permission to read the file. It doesn't matter if the 
remote server in question can not pass the
export check or the user has no read permissions. The source server has 
already authorized read access to that tuple.

  4) The client needs to get a FH on the destination. At this point, it 
will need to pass an export check on
       the destination.
       a) It fails, copy aborted.
           Note - at this point the client should notify the source that 
the tuple is no longer valid.
   5) The client sends a COPY to the destination. At this point, it 
needs to pass a permissions check.
       a) It fails, copy aborted.
       b) It succeeds, it establishes a call back path.
 
   6) The destination now starts to send READ requests to the source. It 
will connect via one of the
       addresses it gave in the COPY_PREP and will connect to one of the 
addresses that the source
       returned in the COPY_NOTIFY.

At this point, the source gets a READ request from the destination. It 
can look at the remote server
address, the FH, and the stateid to determine that normal export and 
permission checks are to be
bypassed.

The stateid does not need to be encrypted in any fashion because it only 
does the magic bypassing
if and only if the READ request is coming from a known address.

Furthermore, if at any time an access cache miss were to occur on either 
server (timeout or the
export was modified), the server can redrive the access check with the 
client address it has cached.
(Needed for the callbacks.)  If the access check fails, the server can 
send a COPY_REVOKE back
to the client. And it in turn can stop the other server.

But note that if it is the destination which does the COPY_REVOKE, it 
will simply stop sending
READ requests to the source. And if it is the source, it can invalidate 
the stateid and stop responding
to the READ requests.

And note that as permission checks are only needed at the first access 
of the files, we won't have to
worry about them changing during the copy process.

If you are worried about someone cloning the stateid and spoofing a 
server, then the answer is the
same as we would give to anyone using AUTH_SYS - move to kerberos.

---

From: James Lentini <[email-masked]>
Date: Fri, 3 Apr 2009 15:34:49 -0400 (EDT)

> expected to be new or updated.

Spencer,

We are waiting for the email comments on the current draft to quiet 
before starting on a new revision. The revisions should take no more 
than 2 weeks once we have all the comments. At the moment, I have 
recorded the following points to address in the next version:

 - record what features of RPCSEC_GSSv3 will be required and how they 
   will be used

 - clarify that operations have a user context (one user cannot abort 
   another users copy)

 - summarize protocol design and rational (diagram, etc.)

 - clarify the interaction with existing AUTH flavors

 - Change instances of "the" server-to-server copy protocol to "a" 
   server to-server copy protocol

 - Explain how NFSv4 may be used as the server-to-server copy 
   protocol

 - note that a cnr_lease_time of 0 corresponds to an infinite lease

 - update COPY_NOTIFY and COPY to contain a list of possible source 
   addresses

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 3 Apr 2009 14:30:27 -0500

> separate discussion altogether. I do not see any details on how that

We're failing to communicate.  I think we should take a step back and
possibly have a conference call, or just wait a while.

But I'll make one last attempt.  Here's what I propose:

C->DST: what name and IP addrs will you use to talk to SRC?
DST->C: [email-masked] and 192.0.2.3

C->SRC: I authorize [email-masked], 192.0.2.3} to READ from FID_src
SRC->C: OK

C->DST: COPY_START(SRC, FID_src, FID_dst)
DST->C: OK

DST->SRC: OPEN FID_src for READ
SRC: <check that DST's name (or IP addrs) is authorized to READ from FID_src>
SRC->DST: OK

DST->SRC: READ(FID_src, offset = 0, size = 8192)
SRC->DST: <data>
DST->SRC: READ(FID_src, offset = 8192, size = 8192)
SRC->DST: <data>
..
..
DST->SRC: CLOSE(...)
SRC->DST: OK

DST->C: CB_COPY_DONE(...)
C->DST: OK

Now, here, any flavor of RPCSEC_GSS will do for the DST<->SRC
communications!  All that needs to happen is that the name for DST that
C told SRC match the name the DST authenticates to SRC as.

AUTH_SYS will work too, if desired as a configuration option -- all that
has to happen is that the IP addresses for DST that C told SRC match the
address that DST uses to talk to SRC with.

It's really quite, quite simple.  I don't understand why you're
complicating things.  Could you please tell my how the above fails to do
what you want?

(Yes, the above requires state at SRC, but it can timeout, in which case
DST's open will fail and C can retry or fallback on client-driven copy.)

Nico
--

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 15:01:38 -0500

...
>  - clarify the interaction with existing AUTH flavors

James,

I think this is now chicken and egg - I don't see the comments starting 
to quiet in the face of
these issues.

Mike mentioned an internal NetApp position on the use of RPCSEC_GSSv3. 
Perhaps it
can be sanitized and shared here?

Thanks,
Tom

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 3 Apr 2009 15:52:36 -0500

> - summarize protocol design and rational (diagram, etc.)

Given the duration to a refresh of the I-D, I would be pleased
to have this item discussed on the WG alias such that we have
consensus here and early.

Spencer

---

From: James Lentini <[email-masked]>
Date: Fri, 3 Apr 2009 17:07:23 -0400 (EDT)

On Fri, 3 Apr 2009, Tom Haynes wrote:

> > We are waiting for the email comments on the current draft to quiet before
> > starting on a new revision. The revisions should take no more than 2 weeks
> > once we have all the comments. At the moment, I have recorded the following
> > points to address in the next version:
> > 
> I think this is now chicken and egg - I don't see the comments 
> starting to quiet in the face of these issues.

To be clear, I wasn't attempting to stop the comments and discussion, 
just answer Spencer's question regarding when a new draft would be 
available.

> Mike mentioned an internal NetApp position on the use of 
> RPCSEC_GSSv3. Perhaps it can be sanitized and shared here?

It would be a start. The best path forward in my opinion is to simply 
continue the discussion.

Anshul posted text from the current RPCSEC_GSSv3 draft that describes 
how privileges can be granted on a per-operation basis.

 http://www.ietf.org/mail-archive/web/nfsv4/current/msg06769.html

I haven't seen an explanation of why this feature could not be 
extended to include a copy privilege. Did I miss someone's reply?

---

From: James Lentini <[email-masked]>
Date: Fri, 3 Apr 2009 17:09:15 -0400 (EDT)

> consensus here and early.

To be clear, I was answering your question on how long a refresh would 
take. I wasn't suggesting that the discussions stop. There have been 
many helpful comments and suggestions on this thread. I appreciate the 
feedback and welcome more.

---

From: "Iyer, Rahul" <[email-masked]>
Date: Fri, 3 Apr 2009 14:29:37 -0700

Comments inline, prefixed by Rahul>

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Friday, April 03, 2009 10:25 AM
To: Madan, Anshul
Cc: [email-masked]
Subject: Re: [nfsv4] server-side copy offload I-D

anshul wrote:
>>>> Why would it be a configuration error to have a private network 
>>>> between the servers?
>>>>     
>>>>         
>>> Good point.
>>>

>>> that source address. By virtue of RPCSEC_GSSv3 privilege, the 
>>> destination will have proven to the source that it is the server 
>> The client presents in the COPY_NOTICE 10.10.20.42 as the address of 
>> the destination server.

Look at the following:

The client sends a COPY_NOTIFY to the source. It sends it to
10.10.20.43, because it has no idea of 10.100.10.43 and can't even talk
to that address. It states to expect a COPY request from 10.10.20.42.

It then sends a COPY to the destination at 10.10.20.42. It states that
the copy is to occur with 10.10.20.43.

How does the destination know that 10.10.20.43 corresponds to
10.100.10.43?
And while I've lined the numbers up, you can't count on that always
happening. And in any event, how does the kernel know this?

10.10.20.43
10.100.10.43

Now the destination sees that it can talk to source at either of the two
addresses. And seeing as how it has interfaces on both of those, it can
chose either. 
We would just
prefer that it uses 10.100.10/24.

For some reason, I'm thinking an implementation would want some form of
a simple policy engine here to make that type of decision. :->

Really, the load might change during the time of day, we might want to
detect that one of the networks is down (or an interface to that network
is down), etc.

Rahul> This sounds reasonable.

Rahul> In traditional AUTH_SYS deployments, the server pretty much
accepts whatever user id the client offers up (root included, depending
on root_squash). That's the current model. Using a similar analogy, if
the source server has a COPY_NOTIFY for the destination, then it should
accept the copy offload request. It doesn't seem any weaker to me. If
the client indeed did want to read data belonging to other users, it
could just as easily do it without jumping through all the offload
hoops. With weak security like AUTH_SYS, why isn't the existence of a
COPY_NOTIFY at the source not sufficient authorization?
Thanks,
Rahul

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 3 Apr 2009 16:24:46 -0500

> extended to include a copy privilege. Did I miss someone's reply?

Yes, you've missed a direct reply from the author of RPCSEC_GSSv3 (hint:
yours truly :)

I'd include a URL to that post, but evidently the archive hasn't updated
yet, but the message ID of that post is [email-masked].

Let's be clear: what RPCSEC_GSSv3 does with privileges is NOT something
that automatically grants the client permission to do something or
other.  RPCSEC_GSSv3 privileges represent the effective privileges
("capabilities" in Linux speak) of the process on behalf of which the
client is acting.  It is up to the server to decide whether to grant the
client-asserted process privileges based on: the maximum privilege that
the server is willing to grant that user on that client.

There is no facility by which a client can delegate authorization in
RPCSEC_GSSv3 as it stands.  Yet as far as I can tell you, Anshul and
Mike think there is or should be.  Moreover, such a facility is NOT
needed, as Tom and I have demonstrated.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 3 Apr 2009 16:36:27 -0500

> yet, but the message ID of that post is [email-masked].

Actually, I just looked at the wrong page of the thread index.  The URL
is:

http://www.ietf.org/mail-archive/web/nfsv4/current/msg06789.html

---

From: Tom Haynes <[email-masked]>
Date: Fri, 03 Apr 2009 22:25:26 -0500

Iyer, Rahul wrote:
> accept the copy offload request. It doesn't seem any weaker to me. 

Yes, the user id presented is accepted and trusted.

But, the server will still check to see if the user id has permissions 
on the directory and/or file.

So, as I understand your argument, I agree that it isn't any weaker. All 
it takes is for a malicious user to
get the right client IP and then send over the right user id to gain access.

> If

The argument that Nico and I present is that COPY_NOTIFY is sufficient 
authorization if the source
keeps track of some state and knows where the server will connect from. 
That is whether we use
AUTH_SYS or RPCSEC_GSSvx.

The source grants authorization and gives the client a stateid. The 
combination of the
(list of destination addresses, FH, stateid)
serve as sufficient authorization for the source to connect. And as a 
matter of fact, we can ignore
the associated userid used by the destination to drive READs.

The kicker in what we present is that now we truly can ignore export 
semantics and use NFSv4.x
operations to do the server-to-server copy.

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 12:41:34 +0530

> So I think you can do the server-to-server copy without using 

That is what our Draft says. With weaker security, we never proposed
that RPCSEC_GSSv3 is required. See the description of COPY_NOTIFY -
 
"DESCRIPTION

   recipient. "

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 12:43:45 +0530

> http://www.ietf.org/mail-archive/web/nfsv4/current/msg06789.html

I think this is a response to a different thread. James was referring to
some other email from us.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 12:44:05 +0530

> AUTH_SYS or RPCSEC_GSSvx.

No, that is the argument that the Draft already presents.  See, I never
received response to this email -
http://www.ietf.org/mail-archive/web/nfsv4/current/msg06769.html. 

Also, See the description of COPY_NOTIFY -

"DESCRIPTION 

RPCSEC_GSSv3 was needed if more security is required to authenticate
destination to source.

>From the beginning that was we were stating. I do not think this is
anything new.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 13:05:07 +0530

> Yes, you've missed a direct reply from the author of RPCSEC_GSSv3 (hint:

Even I missed it.

> RPCSEC_GSSv3 privileges represent the effective privileges
> Mike think there is or should be.  

By extending existing proposal, we feel RPCSEC_GSSv3 can provide
privilege to the particular user of a particular client host to start a
inter server copy on a particular destination server with a particular
source server.

See our counter-arguments to your opposition to our approach (to use
RPCSEC_GSSv3 privileges so that that destination could authenticate
itself to the source), we are still waiting for response to these - 
http://www.ietf.org/mail-archive/web/nfsv4/current/msg06769.html
http://www.ietf.org/mail-archive/web/nfsv4/current/msg06770.html
http://www.ietf.org/mail-archive/web/nfsv4/current/msg06772.html

> Moreover, such a facility is NOT

What is the objective of this approach?

Regards,
Anshul

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Sat, 4 Apr 2009 03:25:06 -0700

> 
> By extending existing proposal, we feel RPCSEC_GSSv3 can 
> provide privilege to the particular user of a particular 
> client host to start a inter server copy on a particular 
> destination server with a particular source server.
> 
> RPCSEC_GSSv3 privileges so that that destination could 
> authenticate itself to the source), we are still waiting for 
> response to these - 

Ok. This part of the e-mail exchange is confusing the heck out of the
already exhausting thread. The whole point of bringing in RPCSECGSS_V3
to extend this to do some futuristic stuff doesn't make sense, when the
whole world is thinking about existing authentication mechanisms and
what protocols could be used for server to server protocols. Can we
address that first? Can I ask you to put extensions to the core of the
proposal on the back burner (for now)?

You are saying that RPCSECGSS_V3 is not required. I (and I'm sure
others) know that. But, what doesn't make sense is why do you not see
that the below is acceptable/workable (or is it just that you are not
following the discussion about COPY_PREP). Your comments about AUTH_SYS
and RPCSEC_GSSv2 being a separate discussion do not ring a bell.

PS: Please do not point any more archives, as this doesn't let the
discussion move forward. If you agree/do not agree with this (that the
below should suffice and server MUST not have to do RPCSECGSS_V3, even
in the context of NFSv4.1), just spell it out.

> What is the objective of this approach?

This is precisely why Nico has been trying to reinforce the same thing
again and again. I don't think Nico is proposing a new approach. He is
highlighting the fact that, thanks to the implementation of COPY_PREP
(new RPC to get the list of names/IP's that destination will use for
read/write) and COPY_NOTIFY, client mediates a basic level of
authorization. Some level of trust is already built in to that model.

Since that level of authorization exists, do we really need to have
server's do (I know it's not MUST) RPCSECGSS_v3. I'd say, it doesn't for
some servers. In that sense, server's could just use existing
authentication methods. Bringing in RPCSECGSS_V3 (NOT REQUIRED it maybe)
also limits the type and kind of server to server protocols that are
used for transfers.

Bottomline, if you think the folks on here need to wait until we get new
text on usage of RPCSECGSS_V3 in the context of the core of this
proposal, please say it. Otherwise, let's split the thread into two
spearate discussions.

- Pranoop

---

From: anshul <[email-masked]>
Date: Sat, 04 Apr 2009 16:46:49 +0530

> address that first? 

I think existing mechanisms are also being discussed.

> Can I ask you to put extensions to the core of the
> You are saying that RPCSECGSS_V3 is not required. 

Yes, not required when we can work with weaker security, in say a
trusted environment. The proposal to use RPCSEC_GSSv3 was in order to
provide a mechanism for the destination to authenticate itself to the
source, thus prove it has required authorization (with help from
COPY_NOTIFY).

Agreed.

> some servers. 

Agreed

> In that sense, server's could just use existing
> authentication methods. 

Agreed. But how does the destination authenticate itself to the source
server, in untrusted environments?

I think Tom and Nicolas are proposing to use RPCSEC_GSS and NFSv4.1 to
solve this. 

Doesn't this restrict the server to server protocols to RPCSEC_GSS +
NFSv4.1?

Regards,
Anshul

> Bringing in RPCSECGSS_V3 (NOT REQUIRED it maybe)

---

From: "Mike Eisler" <[email-masked]>
Date: Sat, 4 Apr 2009 07:13:31 -0700 (PDT)

> consensus here and early.

At the rate the emails are flowing, and the narrow group of participants,
I suspect the majority of the subscribers would be displeased to
deal with say 14 * 50 = 700 more emails. It has certainly exceeded my
ability to process.

(I think this topic is more controversial that discussion in the 1990s to
support 64 bit seconds in the timestamps, which just goes to show,
that when the stakes are at the lowest, the battles are most vicious.)

I think we'd be better off getting a new I-D out that addresses the three
points, rather can continue design by email committee.

BTW, are you writing as a WG chair or as a WG participant? I.e., should the
next I-D be named draft-lentini-nfsv4-... or draft-ietf-nfsv4-... ?

---

From: Spencer Shepler <[email-masked]>
Date: Sat, 4 Apr 2009 10:07:41 -0500

> points, rather can continue design by email committee.

That is fine with me.

> next I-D be named draft-lentini-nfsv4-... or draft-ietf-nfsv4-... ?

As WG co-chair, the next I-D should be draft-lentini-nfsv4...

Spencer

---

From: Tom Haynes <[email-masked]>
Date: Sat, 04 Apr 2009 11:37:57 -0500

> server, in untrusted environments?

The trust is built into the fact that the client tells the source who 
will be the destination.

If the source accepts the client for the copy, it will give it a FH and 
a stateid. The
destination has to present both from one of the addresses it stated it 
would use.
If it presents the request from another address, then it will be rejected.

This is the same level of trust that you have in other AUTH_SYS scenarios.

If you don't trust the stateid and/or list of addresses to be floating 
around, then use the
krb5 variant that builds the trust you want.

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Sat, 4 Apr 2009 12:17:50 -0700

> mechanics of initiating a server-to-server copy and leave out 
> the guts?
> 
> The metadata server-data server protocol is left unspecified. 
> In fact, that's arguably the most important of the 3 
> protocols in question (the client-metadata server and 
> client-data server protocols being the other two). The 
> metadata server-data server protocol is what really makes 
> pNFS work! The other two are just merely scaffolding to 
> expose, in a standardized fashion, the clustered nature of 
> storage systems today. I really don't see why SERVERCOPY 
> isn't any different.

I guess I do not buy this argument that we have a parallel for this in
pNFS specification. The fundamental point is that there are no
well-accepted MDS-DS protocols widely available enough for people to be
aware. Thanks to the fact that MDS-DS protocol is pretty much built into
many of the striped/global file systems, I could see people not caring
much about defining the open semantics for these. Since it's very
closely tied to the filesystem semantics, there is not strong pull/push
to define them as open standard.

However, what this document talks about is a file copy (glorified
transfer) protocol. We have wide variety of these now (e.g ftp, ndmp,
netapp stuff, sun stuff....). While the proposal acknowledges the
server-to-server protocol could be anything, it does not make the data
that the client exchanges with the destination friendly for all
protocols. Meaning, if I want to use NDMP, where is all the necessary
data for the source to initiate the transfer to the destination? Just
sending the IP/hostnames does not suffice. That would make this protocol
very vendor restricted or protocol-restricted (which I think is a
reasonable goal to have for an implementation - not I did not say a
specification) but clearly will not make a number of folks on this alias
happy.

The only parallel we should consider between pNFS and this proposal is
the transport over which the data is shared or exchanged. Meaning, pNFS
allows the blocks, objects, files in the data path. I request that this
should be the fundamental requirement of this proposal.

- Pranoop

> Provide a generic framework and let the two servers negotiate 
> which server-to-server copy method that they are going to use.
> 
>      enum copytype4 {
>              COPY_NFSV4         = 1,
>              COPY_NDMP          = 2,
>              COPY_ZFS_SEND      = 3,
>              COPY_NA_SNAP       = 4
>      };
> 
> request? Doesn't it have to either assume that the servers 
> are different or does it have to be able to query the servers 
> somehow to determine if they are running server-to-server 
> compatible protocol software?
> 
> ---------------------
> 
> BTW:
> 
> Where do we have something like NFS4ERR_PARTNER_PROPRIETARY? 
> I.e., how does a server give back an error code to the client 
> that states it does support a remote server from vendor A but 
> not vendor B?
> 
> ----------------
> 
> 
> 
> Or is NetApp going to provide a na-cp command for every 
> NFSv4.x client implementation?
> 
> one responsible for determining if the other server is 
> compatible. And perhaps this is really a bitmask...
> 
> 
> My concern here is that we are defining part of the 
> framework, saying that heterogeneous server-to-server copy is 
> too hard, and are going to get burnt when a user does use try

---

From: Tom Haynes <[email-masked]>
Date: Sat, 04 Apr 2009 16:28:20 -0500

Erasani, Pranoop wrote:
> protocols. 

Can it?

I'll admit my main interest in this model is to make sure that we can 
define a server-to-server
copy which is totally based within NFSv4.x.

If we back up a bit, the current COPY_NOTIFY states that it works on a 
FH. That would
give the user at the source (via RPCSEC_GSSv3) the privilege of reading 
from that FH.
How many of these transfer protocols work on a FH?

I.e., they will need to perform other operations in order to get to that 
FH. Is the proposal to
allow that authenticated user to do this other stuff?

> Meaning, if I want to use NDMP, where is all the necessary

I think we are agreeing here, but my point is that even the FH does not 
suffice.

> specification) but clearly will not make a number of folks on this alias

Perhaps the only thing we can do here within the confines of NFSv4 is 
define how a server-to-server
copy would proceed with just NFSv4.x procedures.

I.e., a general framework for allowing an arbitrary protocol to initiate 
a server-to-server copy sounds
like it is outside the scope of the NFSv4 protocol.
> should be the fundamental requirement of this proposal.

I agree - I don't think the COPY should just work on a file. Perhaps you 
can suggest how to modify
the RPC data structures here? :->

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Sat, 4 Apr 2009 16:20:09 -0700

> > that the client exchanges with the destination friendly for all 

Probably could. If enough effort is put into separating the definition
of a file server, definition of unit of file copy, definition of
framework where s2s protocol-independent information could be exchanged
(and etc..), then it could be done. It's not so easy to define something
that encompasses all set of file transfer/copy protocols.

Yes, I'm also unduly picking on the title and highlighting that the
proposal doesn't have all it takes to support the title.

> 
> I'll admit my main interest in this model is to make sure 
> that we can define a server-to-server copy which is totally 
> based within NFSv4.x.
> 
> If we back up a bit, the current COPY_NOTIFY states that it 
> works on a FH. That would give the user at the source (via 
> RPCSEC_GSSv3) the privilege of reading from that FH.

Some, I guess. Even if they work on a FH, the concept of FH as defined
in the confines of NFS specification (supposed to be opaque) may not
work for other protocols.  FH as such could represent information on the
file from protocol perspective, but if the guts of transfer happens over
other protocols, I'd say the FH may not make sense. So, we need some
sorta translation layer that interprets the NFS protocol FH to s2s
protocl FH. OTOH, what would be the story with heterogenous vendors as
source and destination?

> 
> I.e., they will need to perform other operations in order to 
> get to that FH. Is the proposal to allow that authenticated 
> user to do this other stuff?

Why stretch this all the way to authentication? How about source and
destination being heterogenous? Is this proposal gonna allow such
translation layers?

> I think we are agreeing here, but my point is that even the 

Well taken and aptly describes the basics of the confusion.

> like it is outside the scope of the NFSv4 protocol.

Yes. That's fine. Please call this "NFSv4 server-to-NFSv4 server copy
offload". Then, I'd shutup on this thread and focus on NFSv4'ish
protocols then.

> the RPC data structures here? :->

That would be a whole new different document/proposal. ;) My interest in
this is to make sure it meant what it said.

- Pranoop

---

From: Tom Haynes <[email-masked]>
Date: Sat, 04 Apr 2009 20:37:21 -0500

>  - clarify that operations have a user context (one user cannot abort 

Actually, it can't just be a per user and a per file context, it has to 
be an instance context.

Say the user does:

% cp /net/source/big-file.iso /net/destination/archives/big-file.iso &
% cp /net/source/big-file.iso /net/destination/alpha/big-file.iso &

and they want to revoke the first and not the second.

Normally we'd view that the source server had the same incore inode 
being shared between
the two COPY instances (with a refcnt of 2). And since the destination 
server and FH matched,
the server might even hand out the same stateid here.

But I'm going to argue that the source server has to hand out 2 distinct 
identifiers here.

That way the client can do a COPY_REVOKE on either one and have the 
other finish.

If it is just per user and per file, then the COPY_REVOKE will end both 
instances and you
will surprise the user.

---

From: anshul <[email-masked]>
Date: Sun, 05 Apr 2009 13:26:26 +0530

> If the source accepts the client for the copy, it will give it a FH and 

I think the source should not to give a stateid. The stateid should be
given only when the COPY operation starts.

> krb5 variant that builds the trust you want.

I think that restricts you to use NFSv4.x as the server-to-server copy
offload protocol. 

Regards,
Anshul

---

From: Tom Haynes <[email-masked]>
Date: Sun, 05 Apr 2009 13:44:59 -0500

> given only when the COPY operation starts.

How else do you describe the authorization of the COPY_NOTIFY?

I've argued that each COPY_NOTIFY needs to return an unique identifier 
to some per instance
state on the source server. If the same user initiates two copies from 
the same source file to
different locations on the same destination server, how can one of those 
be COPY_REVOKE
without an unique identifier?

Also, when the COPY operation starts, in terms of NFSv4.x, the 
destination server can start
issuing READ operations right away. What does a READ operation need?

(cfh), stateid, offset, count -> eof, data

So, the COPY operation needs to present a stateid for NFSv4.x 
server-to-server offload
to proceed. Perhaps each s2s protocol will impose another needed 
parameter. Which is
why I question whether we can do a general framework within the confines 
of NFSv4.x
or should we just concentrate on allowing NFSv4.x to be the s2s offload 
protocol?

Also, note that the server-to-server offload protocol is limited to only 
operations which READ
to that FH.

Another interesting question here is that the destination server is in 
effect a client of the
source server in this instance. Does it need to establish a clientid?

I.e., that all goes back to Pranoop's question about whether the 
destination needs to do a
handshake with the source and the question I posed the other day about 
whether the
client (and/or destination) needs to have a read delegation on the source.

And the other open question I had was whether each server has to 
establish a callback
path to the server?

> I think that restricts you to use NFSv4.x as the server-to-server copy

Before you can start to use anything user than NFSv4.x as the 
server-to-server copy offload
protocol, you need to show how that other method is authorized to access 
the source server.

The COPY_NOTIFY simply states that the destination is authorized to read 
from the source
FH. It does not authorize the destination to read from the underlying 
filesystem. The impact
of that is that the destination may not be able to do a path traversal 
to drive a file open.

Within the confines of what is currently proposed, the only 
server-to-server copy offload
protocols that can work are those that can handle a FH. And I think that 
is too restrictive
for most other protocols. You are going to have to heavily hack them up 
to get this to work.

> Regards,
> Anshul
>
>   
>>> I think Tom and Nicolas are proposing to use RPCSEC_GSS and NFSv4.1
>>>       
>> to
>>     
>>> solve this. 
>>>

---

From: "Mike Eisler" <[email-masked]>
Date: Sun, 5 Apr 2009 12:21:15 -0700 (PDT)

I was hoping this thread would go comatose until James push -01 of I-D.
But I have to act counter to my suggestion to not send more email until
-01, because of an apparent misunderstanding.

> How else do you describe the authorization of the COPY_NOTIFY?

COPY_NOTIFY privilege is indicated by soure fh, source NFS server
(implied by the target of COPY_NOTIFY) and destination NFS server.

> without an unique identifier?

Send COPY_REVOKE to the source to revoke authority, which affects
all copies from a specified source server and filehandle pair to
a specified destination.

Send COPY_ABORT to the destination to abort a copy in progress.

> filesystem.

A distinction without meaning.

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Apr 2009 00:10:43 -0500

Michael has provided some logistical information for the June BakeAthon 
at NetApp Sunnyvale:

http://www.connectathon.org/bakeathon/

I'll update that page as further details develop.

---

From: Tom Haynes <[email-masked]>
Date: Sun, 05 Apr 2009 23:48:48 -0500

> (implied by the target of COPY_NOTIFY) and destination NFS server.

I should have used the word "record" instead of "describe".

How does the source record an authorization?  I know you are arguing 
that it doesn't need to
do so, but I am arguing that it needs to be able to differentiate users 
(which means it needs to
store the user that initiated the COPY_NOTIFY) and instances with a 
given user (which
means it needs to store an id).

> all copies from a specified source server and filehandle pair to

Again, this has the unexpected side effect that revoking one s2s copy 
session will revoke others.

And unless you do add a user context, it will mean that one user can 
revoke an other user's session.

> A distinction without meaning.

No, it has a lot of meaning. What did you state as the authorization for 
a s2s copy above?

source FH, source server, destination server

That means that the FH of the directory entry containing the file to be 
copied is not
sufficient for authorization.

Ahh, I see how you missed my point - I wasn't clear enough.

The COPY_NOTIFY process proceeds by the client traversing the source's 
namespace until
it finds the file it wants to copy. I.e., before the COPY_NOTIFY 
procedure can be invoked,
it must first establish a FH in memory. Once it does that, the 
COPY_NOTIFY is performed.

If it succeeds, then the client has an authorized FH it can send to the 
destination.

When the destination starts the s2s copy, it can not assume that it also 
can traverse the
source's namespace. This is what I meant by not being able to read from 
the underlying
filesystem. I.e., the client has read privileges to the source's 
filesystem, but the
destination does not.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 6 Apr 2009 00:05:27 -0500

> server, in untrusted environments?

Using Kerberos V or whatever you have.

> I think Tom and Nicolas are proposing to use RPCSEC_GSS and NFSv4.1 to

Yes, though, of course, RPCSEC_GSSv3 would be fine too.

> Doesn't this restrict the server to server protocols to RPCSEC_GSS +

Why would you think that?  You can have a proprietary server-to-server
copy protocol if you like.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 6 Apr 2009 00:13:09 -0500

> inter server copy on a particular destination server with a particular

I still don't quite know what you mean by "privilege."  What you write
above makes no sense unless I replace "privilege" with "authorization."
See below.

> http://www.ietf.org/mail-archive/web/nfsv4/current/msg06772.html

I did respond.  This thread is far too long now.  Tom and I have made
very a specific proposal that requires no fancy new features at the
RPCSEC_GSS layer and you're still not answering it.

> What is the objective of this approach?

To keep RPCSEC_GSSv3 simple.  Specifically: I don't want to add anything
about delegation of authorization (note: authorization) to RPCSEC_GSSv3
if we can avoid it because I think that sort of thing belongs at the
application layer in this case.

Nico
--

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 12:17:46 +0530

> >> without an unique identifier?

We do not want to revoke a particular instance, we want to revoke all
instances of the given combination of source server, source fh,
destination server. To revoke a particular instance, the client can
issue COPY_ABORT to the destination server.

See the description of COPY_REVOKE - the problem we expect COPY_REVOKE
to solve :
"The COPY_REVOKE operation is useful in situations involving an
   asynchronous server-to-server copy.  For example, suppose an NFS
   client initiates a copy of file X from NFS server A to NFS server B.
   If NFS server B decides to perform an asynchronous copy and a network
   partition prevents communication between the NFS client and NFS
   server B (without disrupting communication between the NFS client and
   NFS server A or between NFS server A and NFS server B), the NFS
   client may like to write new information to file X. In this
   situation, the client may treat the copy as a failure and start
   writing to file X regardless of the ultimate status (if the network
   partition is removed and the copy operation does succeeded, the
   contents of the copied file would be uncertain).  However, if the
   client wishes to write sensitive information to file X that NFS
   server B is not intended to see, the client MUST use a COPY_REVOKE
   operation to ensure NFS server B is prevented access to the updates
   to file X"

The user should be able to revoke the all the copies, each to different
locations.

> source's namespace. 

The destination does not need to traverse the source namespace, it
already has the desired source FH.

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 12:18:14 +0530

> copy protocol if you like.

As per your proposal - there is a caveat - that the propriety protocol
would have to use RPCSEC_GSS, agreed?

Regards,
Anshul

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 12:18:54 +0530

> above makes no sense unless I replace "privilege" with "authorization."

Ok, let us kill this thread. And continue this discussion when we come
up with the new I-D.

>  Tom and I have made

I think we are answering it. See my reply in the previous email.

Regards,
Anshul

---

From: "Muntz, Daniel" <[email-masked]>
Date: Mon, 6 Apr 2009 00:03:58 -0700

The s2s copy protocol can be any protocol as long as it can use an
NFSv4.x fh on the destination server (obtained from the source server)
to initiate a copy from the source server. Hmm... IMHO that pretty much
implies that the s2s copy protocol is either NFSv4.x or a homogeneous
cluster that knows how to unroll an fh. I suppose one could argue that
the fh could be used in an arbitrarily complex protocol to allow the
destination to obtain some other representation of the source file from
the source server by starting from the fh (e.g., fh to pathname), but
that would be silly.

  -Dan

> Sent: Sunday, April 05, 2009 12:56 AM
> To: Tom Haynes
> Cc: [email-masked]; Madan, Anshul
> > If the source accepts the client for the copy, it will give it a FH 
> > and a stateid. The
> 
> I think the source should not to give a stateid. The stateid 
> should be given only when the COPY operation starts.
> 
> > 
> I think that restricts you to use NFSv4.x as the

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 12:51:54 +0530

On Mon, 2009-04-06 at 12:33 +0530, Muntz, Daniel wrote:
> to initiate a copy from the source server. 

Yes.

My reply was to Nicolas proposal to use RPCSEC_GSS between source and
destination. If RPCSEC_GSS is the only mechanism we have to authenticate
the destination to the source, I think that restricts us to use s2s
protocols that work with RPCSEC_GSS.

Regards,
Anshul

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Mon, 6 Apr 2009 00:46:00 -0700

> Sent: Monday, April 06, 2009 12:22 AM
> To: Muntz, Daniel

Do we expect that assertion to showup in the next revision of the
document in fine-print?

Currently, the document says:

   If the source object and destination object are on different file
   servers, the file servers will communicate with one another to
   perform the copy operation.  The server-to-server protocol by which
   this is accomplished is not defined in this document.

Effectively, with your comment, a basic requirement of such
server-to-server protocol seems to be defined. I'l opine that this
document would have made much sense, if that was clarified at the
outset.

> My reply was to Nicolas proposal to use RPCSEC_GSS between 
> source and destination. If RPCSEC_GSS is the only mechanism 
> we have to authenticate the destination to the source, I 
> think that restricts us to use s2s protocols that work with 
> RPCSEC_GSS.

I think security is of the least concern. The fact that you have to use
NFSv4.x fh on the destination rules out any protocol that needs a name
to initiate the copy (think FTP). And I think this proposal adds very
less value for heterogenous environments.

- Pranoop

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 13:21:22 +0530

> Do we expect that assertion to showup in the next revision of the

Ok. We will clarify this.

> document would have made much sense, if that was clarified at the

Ok.

> to initiate the copy (think FTP). 

Agreed.

Regards,
Anshul

> And I think this proposal adds very

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Mon, 6 Apr 2009 00:57:23 -0700

Given that you agree (not sure all of the co-authors agree to this yet),
I suggest changing the title to NFS server-to-NFS server copy protocol.
That would take a lot of angst out of the feedback, as most of the
questions in here are around the heterogenous and non-NFS environments.

- Pranoop

> > > My reply was to Nicolas proposal to use RPCSEC_GSS between source 
> > > and destination. If RPCSEC_GSS is the only mechanism we have to 
> > > authenticate the destination to the source, I think that 
> restricts 
> > > us to use s2s protocols that work with RPCSEC_GSS.
> > 
> > I think security is of the least concern. The fact that you have to 
> > use NFSv4.x fh on the destination rules out any protocol 
> that needs a 
> > name to initiate the copy (think FTP).
> 
> Agreed.
> 
> Regards,
> Anshul
>

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 13:58:38 +0530

Just to clarify -

> > > name to initiate the copy (think FTP).

I agreed that s2s protocols that need a name cannot be used, but disagree that security is of least concern.
Also on second thoughts, if there is a reasonable way to resolve the source NFS4.x fh to a name, then even s2s protocols requiring a name could be used.

> questions in here are around the heterogenous and non-NFS environments.

Since NFS operations are required to be issued to both source and
destination servers, we never expected anyone to use these in non-NFS
environments.

Regards,
Anshul

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Mon, 6 Apr 2009 01:45:32 -0700

> I agreed that s2s protocols that need a name cannot be used, 
> but disagree that security is of least concern.

I think I was making a point that if the fundamentals are not addressed
(the whole confusion about non-NFS protocols), then security can be put
in the backburner until that is addressed.

> Also on second thoughts, if there is a reasonable way to 

I though the whole point of a IETF spec is to describe a reasonable
generic way. ;)

> resolve the source NFS4.x fh to a name, then even s2s 
> protocols requiring a name could be used.

In an environment where any liberty can be taken, it doesn't take much
to do that.

However, we are talking on [email-masked], where we need to see, if the
current prposal is deficient in supporting such environemts. If not, we
have to figure out what can be added to make it work. Clearly, without
standardizing the proposed RPC procedures to better accomdate such s2s
protocols, I don't see why the existing widely used s2s protocols can be
used without writing new hacky messages/protocols on top of them.
Meaning, I can't expect FTP to go and get me the name given the
filename. I'd have to build another layer/protocol on top of FTP to do
the translation. Given I'd have to write a layer, why would I have to
use NFSv4.x to initiate such transfer in the first place. :(

Looking at the basics of the proposal, neither I think there was an
intention, nor I feel this could be easily extended to accomadate all
sorts of s2s protocols. If you disagree, I'l continue to disagree as
well.

- Pranoop

> > Given that you agree (not sure all of the co-authors agree to this 
> > yet), I suggest changing the title to NFS server-to-NFS 
> server copy protocol.
> Since NFS operations are required to be issued to both source 
> and destination servers, we never expected anyone to use 
> these in non-NFS environments.
> 
> Regards,
> Anshul
> 
> > 
> > - Pranoop
> > 
> > > > > source and destination. If RPCSEC_GSS is the only 
> mechanism we 
> > > > > have to authenticate the destination to the source, I 
> think that
> > > restricts
> > > > I think security is of the least concern. The fact that 
> you have 
> > > > to use NFSv4.x fh on the destination rules out any protocol
> > > that needs a
> > > > And I think this proposal adds very less value for heterogenous 
> > > > environments.
> > > > - Pranoop
> > > > 
> > > > > 
> > > > > Regards,
> > > > > Anshul
> > > > > 
> > > > > >   -Dan
> > > > > 
> > > > >

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 15:03:14 +0530

> I though the whole point of a IETF spec is to describe a reasonable

Right, the s2s protocol needs to provide that, but defining the s2s
protocol was not the original intention of this draft.

> used without writing new hacky messages/protocols on top of them.

What sort of generic extensions to the proposed RPC procedures you think
are need to accomodate existing s2s protocols? 

> filename. 

Not FTP, but the server implementations can take care of such
translations.

> I'd have to build another layer/protocol on top of FTP to do

Without offloading copying to the server, an NFS client would have to
read all data from the source server , then write all data back to the
destination server. With this proposal, it can just issue an RPC to
offload the whole thing to the server.

> sorts of s2s protocols. 

Right, the intention was not to accomodate all sorts of existing s2s
protocols, though those that can be extended would be relevant here.

> If you disagree, I'l continue to disagree as
> well.

Ok.

Regards,
Anshul

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 6 Apr 2009 07:55:50 -0400

> that it doesn't need to do so, 

I don't think anyone is arguing that the source doesn't need to record
the authorization, and I don't see how they could argue that.

If I authorize through COPY_NOTIFY (or not) and then there is a
subsequent COPY, it is either going to succeed or fail depending on
whether the authorization was previously done (or not).  That requires
that the fact of the whether the authorization occurred needs to be
recorded, since without it being recorded, you can't make that decision.
QED.

> but I am arguing that it needs to be able to differentiate users 
> (which means it needs to store the user that initiated the
COPY_NOTIFY) 

I agree with that.

> and instances with a given user (which means it needs to store an id).

It would need to store some identification of the user but the question
(I think) is whether it needs to return an id to the client doing the
COPY_NOTIFY, and so far, I haven't seen any indication that it does.  It
may have an internal id but that's an implementation detail that need
not and should not be mentioned.

Returning an id to the caller adds nothing.  Without an id I have to
check that the right user is revoking the authorization but with an id I
still have to check that.

-----Original Message-----
From: Tom Haynes [mailto:[email-masked]] 
Sent: Monday, April 06, 2009 12:49 AM
Cc: [email-masked]
Subject: Re: [nfsv4] server-side copy offload I-D

Mike Eisler wrote:
> But I have to act counter to my suggestion to not send more email 
> until -01, because of an apparent misunderstanding.
>

>>>> and a stateid. The
>>>>
>>>>         
>>> I think the source should not to give a stateid. The stateid should 
>>> be given only when the COPY operation starts.
>>>
>>>
>>>       

How does the source record an authorization?  I know you are arguing
that it doesn't need to do so, but I am arguing that it needs to be able
to differentiate users (which means it needs to store the user that
initiated the COPY_NOTIFY) and instances with a given user (which means
it needs to store an id).

>> I've argued that each COPY_NOTIFY needs to return an unique 
>> identifier to some per instance state on the source server. If the 
>> same user initiates two copies from the same source file to different

>> locations on the same destination server, how can one of those be 
>> COPY_REVOKE without an unique identifier?
>>     
>

a s2s copy above?

can traverse the

---

From: "Mike Eisler" <[email-masked]>
Date: Mon, 6 Apr 2009 07:28:13 -0700 (PDT)

COPY_NOTICE, COPY, COPY_ABORT, COPY_REVOKE all operate with the
identity of user principals like every other operation in NFSv4 that
uses a current filehandle.

This stuff about privileges to access a file system alludes to
a protocol that is not NFSv4.

On Sun, April 5, 2009 9:48 pm, Tom Haynes wrote:
>>>>> and

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 6 Apr 2009 09:45:58 -0500

> would have to use RPCSEC_GSS, agreed?

No, there is not.  Why would you think that there is??

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 6 Apr 2009 09:58:22 -0500

> No, there is not.  Why would you think that there is??

In fact, let me go one step further:

    If you build the server-side copy operations the way Tom and I
    proposed then you can do anything you like for the server-server
    protocol.  On the other hand, if you insist on making RPCSEC_GSSv3
    do unnatural things then everyone will be forced to use RPCSEC_GSSv3
    for the server-server protocol.

All you have to do is read our proposal and walk through it.

Nico
--

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Apr 2009 10:50:54 -0500

anshul wrote:
>    contents of the copied file would be uncertain).  

The above talks about revoking a specific instance of a write. Please 
rewrite the example
here in COPY_REVOKE to provide a rational as to why a client would want 
to do a COPY_REVOKE.

And the following clearly states that a COPY_REVOKE is to be used to 
stop a single instance of
a user writing to a specific file.

> However, if the

Move this to the COPY_ABORT section and this makes more sense.

Also, it would seem to me that in this example here, where the client 
wants to start writing to
the destination FH, you would want to issue a COPY_ABORT to the 
destination and not
a COPY_REVOKE to the source.

I.e., the destination will have to deal with two different processes 
trying to write to the
same file. I would want to make sure that one of those processes was 
halted before the
next started.

> already has the desired source FH.

You miss my point altogether - to make a random s2s offload protocol 
work, it has to use the
FH. So do there exist such applications which do not use FHs? Are you 
going to rewrite them
such that they can handle FHs?

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Apr 2009 11:11:31 -0500

Noveck, Dave wrote:
> COPY_NOTIFY, and so far, I haven't seen any indication that it does. 

Yes, I agree with that statement, it does not need to return the id.

>  It

And I agree with that.

> Returning an id to the caller adds nothing.  

I disagree with this, in respect that it makes a NFSv4.x s2s copy much 
easier. And I'll show
that at the bottom of this email....

> Without an id I have to

Yes, I agree with that.

BTW: given the bad example in the draft for COPY_REVOKE, if a client 
wishes to revoke
a specific copy instance on the source, how does it identify that instance?

Why a stateid could be good:

When you read a file via NFv4, it can be abstracted as:

open the file (gets the FH)
while (data in FH)
    read FH

Excuse my poor pseudo-code syntax.

What does the READ operation need? FH and stateid.

So, a destination doing a COPY from a source, could be abstracted as:

get a COPY (provides FH)
while (data in FH)
    read FH

But where does the destination get its hand on a stateid?

Okay, in a normal read from a file, the OPEN provides the stateid. When 
the client does the
COPY_NOTIFY, does it OPEN the file? It doesn't have to.

If it does, we could provide the destination with that stateid.

If it doesn't, then we can modify the COPY_NOTIFY to return a stateid.

---

From: anshul <[email-masked]>
Date: Mon, 06 Apr 2009 21:46:55 +0530

> a user writing to a specific file.

No, I think you are missing the purpose stated in the example
altogether. If a single instance is revoked, other instances of copy
operations would still allow the destination to read sensitive
information, that the client starts to write to file X after it has
COPY_REVOKEd the file X.

Also see the first para of the description - 

"This operation revokes a remote server's ability to copy data from
   the recipient.  It instructs the recipient to prevent a copy
   operation from the server identified by cra_destination_server on the
   file specified by CURRENT_FH.  If the cra_destination_server has
   already begun copying the file, a successful return from this
   operation indicates that further access will be prevented."

Here it is clearly stated that the intention is to "revokes a remote server's ability to copy data from 
the recipient", NOT an revoking a single instance of a user writing a file. That does not make sense at all, given the purpose.

> Move this to the COPY_ABORT section and this makes more sense.

Why? 

> the destination FH, 

I do not see how. It is clearly stated that file X is on the source and
client intends to write to file X.

Regards,
Anshul

> you would want to issue a COPY_ABORT to the 
> destination and not

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Apr 2009 11:38:09 -0500

> information, that the client starts to write to file X after it has

In general, if you write something down and someone else has trouble 
reading it, does that
imply that the reader is missing something or that the writer has not 
been clear enough? In
a murder mystery, that might be fun. In a spec, it is dangerous.

This line is confusing:

> In this
>    writing to file X regardless of the ultimate status

I read this as the client wants to write to file X on the destination. 
Evidently your intention
here is that the client wants to write to file X on the source. You 
could state that explicitly.

---

From: Tom Haynes <[email-masked]>
Date: Mon, 06 Apr 2009 12:56:12 -0500

> a protocol that is not NFSv4.

I'm glad that we violently agree that the proposed COPY operations all 
use the identity of
user principals and the current filehandle. And that those user 
principals do not mean squat
if the current filehandle is not the one authenticated by the source server.