Subject: Re: [nfsv4] NFS4.2 inter-server copy spec concern

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 3 Mar 2015 17:49:26 -0500

On Fri, Feb 27, 2015 at 12:02 PM, Tom Haynes
<[email-masked]> wrote:
>
> On Feb 26, 2015, at 7:43 AM, Olga Kornievskaia <[email-masked]> wrote:
>
> Hi folks,
>
> I'd like to raise a concern that the current specification in COPY_NOTIFY is
> insufficient (or maybe that it makes implementation unnecessary complicated
> on the servers).
>
> At the time of COPY_NOTIFY, the source server needs to remember
> cna_src_stateid and match it to the corresponding READ request from the
> destination server.
>
> My first concern is about matching ips. At the time of COPY_NOTIFY, the
> source server has the destination server ip provided by the client, using
> the spec example say it's 203.0.113.56. Thus the source server can store say
> cna_src_stateid=0xf00bar and cna_destination_server=203.0.113.56. Yet, the
> READ can come from 192.0.2.56 which won't match any stored values. The only
> thing that source server can check is that READ came from the same subnet,
> 192.0.2.0/24 for which it included its ip of 192.0.2.18 in the
> cnr_source_server list in the reply of COPY_NOTIFY. Thus, I don't see how's
> possible to verify that READ request came from the destination server the
> client specified in COPY_NOTIFY (unless it's mandated that the destination
> server MUST use the same ip used with the client).
>
>
> Look at Section 4.10.1.2:
>
>    When the client sends the source server the COPY_NOTIFY operation,
>    the source server may reply to the client with a list of target
>    addresses, names, and/or URLs and assign them to the unique
>    quadruple: <random number, source fh, user ID, destination address
>    Y>.  If the destination uses one of these target netlocs to contact
>    the source server, the source server will be able to uniquely
>    identify the destination server, even if the destination server does
>    not connect from the address specified by the client in COPY_NOTIFY.
>    The level of assurance in this identification depends on the
>    unpredictability, strength and secrecy of the random number.
>
> I.e., in this case it is the “secret number” which authenticates the
> destination, even from a different IP than provided.
>
>
> My second concern is regarding uniqueness of stateid. There is only a
> guarantee of uniqueness of stateids for a clientid. Two different clientids
> that open the same file can get the same stateid. Thus, if we have two
> copies initiated from two different clientids, then the source server is
> required to deal with non-uniqueness by either ref counting or allowing for
> duplicates on the list.
>
>
> The concern is that the copies are from different clients, from the same
> source, and to the same destination?
>
>
> If gssv3 is used then we can use the generated shared secret to provide us a
> lookup key for stateids to be matched. However, spec allows for non-gssv3
> implementation. Using user identity (i.e., user@nfsdomain) as a key also
> doesn't seem to work because the destination server will be making the READ
> as user@nfsdomain in gssv2. Maybe ref counting or duplicate entries is a
> perfectly fine implementation but seems like if we added a random_number to
> COPY_NOTIFY (as is suggested for things like urls) it would make
> implementation must easier.
>
> Thank you.
>
>
>
> Why can’t the source use that random number it generated and gave to the
> client as a key? So the burden would be on the source to not generate the a
> random number already in use.

I think without a random number the structure there will never have
use for NL4_NETADDR type in the netloc4 argument of COPY_NOTIFY. I
think it's makes more sense to change the COPY_NOTIFY4args to include
the random number. It's not possible to satisfy all the checks
requirement and only send a stateid and netaddr in the COPY_NOTIFY.

---

From: Tom Haynes <[email-masked]>
Date: Tue, 3 Mar 2015 15:28:19 -0800

> On Mar 3, 2015, at 2:49 PM, Olga Kornievskaia <[email-masked]> wrote:
> 

I’m not convinced of the real utility of such a random number. If an attacker is
able to determine the FH and stateid, they can get at the random number as well.

Also, timing wise, the draft has passed working group last call and is waiting to be
sent to the IESG. There are also implementations out there.

Ask on the Linux NFS WG if they are adverse to taking such a change?

But please do it soon, tick-tock, tick-tock. :-)

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 3 Mar 2015 19:02:40 -0500

> able to determine the FH and stateid, they can get at the random number as well.

I'm not looking at this as a security issue (spec already suggested
that COPY_NOTIFY should be protected with GSS privacy if no gssv3 is
used). I'm looking at this a server implementor for the spec
requirement that the source server must verify that READ's stateid is
the one that the source server gave out to in reply of the
COPY_NOTIFY. One concern is that the source server can't check the
requestor's (destination server's) ip address as it's allowed to have
a multi-homed server. Secondly, the stateid issues has no uniqueness
properties.

> sent to the IESG. There are also implementations out there.

Ok. I was asking this question with hopes of the receiving guidance
from the IETF community to the server implementors with regards to
meeting the requirements of the spec. If somebody else can please
share how it's possible, I'd really like to know.

> Ask on the Linux NFS WG if they are adverse to taking such a change?

by "WG" do you mean the mailing list? I can surely do that but I
believe there are no linux implementation of server copy. We are
currently working on that and facing this issue and therefore asking
about it.

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Olga Kornievskaia <[email-masked]>
Date: Tue, 3 Mar 2015 19:08:40 -0500

>> But please do it soon, tick-tock, tick-tock. :-)

Also forgot to ask, I thought the draft is not yet published (rfc)
therefore I'm raising this as a spec issue that I think is of major
concern that should be fixed before the draft becomes unchangable. Or
else, I'd like my faulty reasoning be corrected.

---

From: Olga Kornievskaia <[email-masked]>
Date: Wed, 4 Mar 2015 10:07:22 -0500

On Wed, Mar 4, 2015 at 9:49 AM, J. Bruce Fields <[email-masked]> wrote:
> On Tue, Mar 03, 2015 at 03:28:19PM -0800, Tom Haynes wrote:
>>
> Of server-to-server copy?  Which ones?

Inter-server.

> --b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 4 Mar 2015 09:49:35 -0500

--b.

---

From: Tom Haynes <[email-masked]>
Date: Wed, 4 Mar 2015 10:18:14 -0800

> On Mar 3, 2015, at 4:02 PM, Olga Kornievskaia <[email-masked]> wrote:
> 
> 
> COPY_NOTIFY.

There is no stateid given back as part of the COPY_NOTIFY.

> One concern is that the source server can't check the

Basically, you are asking for a random number which is in effect a
global (across all clients) identifier for copy state on the server. Which
sounds exactly like a stateid to me.

So instead of adding a random number I would propose a copy stateid
which is unique across all clients.

Add to 15.3.3:

   struct COPY_NOTIFY4resok {
           nfstime4        cnr_lease_time;
           stateid4        cnr_stateid;
           netloc4         cnr_source_server<>;
   };

   The cnr_stateid is a copy stateid which uniquely describes the
   state needed on the source server to track the proposed copy.
   While normal stateids are only gaurenteed to be unique per client
   ID, the cnr_stateid is unique across all client IDs. The source
   server may implement this by reserving the high order bit of all
   stateids as indicating that the stateid is a copy stateid.
   This copy stateid MUST equate to the each of earlier
   delegation, locking, and open states for the client on the
   given file (see Section 4.4.1).

Modify in 15.2.3:

   For an intra-server copy, both the ca_src_stateid and ca_dst_stateid
   MUST refer to either delegation, locking, or open states provided
   earlier by the server (see Section 4.4.1).  The order of selection is
   explained in Section 8.2.5 of [RFC5661].  If either stateid is
   invalid, then the operation MUST fail.  If the request is for a
   inter-server copy, then the ca_src_stateid can be ignored.  If
   ca_dst_stateid is invalid, then the operation MUST fail.

to be:

   For an intra-server copy, the ca_dst_stateid MUST refer
   to either a delegation, a locking, or an open state provided
   earlier by the destination server.  And the ca_src_stateid
   MUST be the cnr_stateid returned from the earlier COPY_NOTIFY.

But to take this change, we need broad agreement that this new
form of stateid is acceptable.

---

From: "Adamson, Andy" <[email-masked]>
Date: Wed, 4 Mar 2015 18:37:15 +0000

> On Mar 4, 2015, at 1:18 PM, Tom Haynes <[email-masked]> wrote:
>
>

From 5661:

   A stateid with the same bit pattern for one client may designate an
   entirely different set of locks for a different client.

so a stateid is not unique global identifier for copy state on the server, as the same stateid bit pattern can exist under different clientids.

As 4.2 inter-server copy stands now, the READ from the destination server (acting as an NFS cllient) to the source server sends the ca_src_stateid from the COPY (the OPEN sateid from the client) but the source server has no way to tell which clientid the stateid is associated with. Thus the source server can not tell the difference between two copies from different clients from the same source server to the same destination server where the stateid’s from each of the two client OPENs have the same bit pattern.

—>Andy

> form of stateid is acceptable.

The information contained in this message and in any attachments is privileged and confidential, and may contain personal data. It is intended only for the use of the addressee. You should not read, copy, use, or disclose this information without authorisation. If you are not the intended recipient of this message, please e-mail the sender immediately and delete this message. Any unauthorised dissemination, disclosure, copying or use of the contents of this message is strictly prohibited and may result in legal action, including civil or criminal proceedings.

NetApp-Disclaimer-v1.1

---

From: Tom Haynes <[email-masked]>
Date: Wed, 4 Mar 2015 10:46:48 -0800

> On Mar 4, 2015, at 10:37 AM, Adamson, Andy <[email-masked]> wrote:
> 
>> 

Right, which is why I defined it below to be unique across all client IDs.

>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
> 
> 
>

---

From: "Adamson, Andy" <[email-masked]>
Date: Wed, 4 Mar 2015 19:44:52 +0000

> On Mar 4, 2015, at 1:46 PM, Tom Haynes <[email-masked]> wrote:
> 
>> 

So it’s not a stateid! Call it something else.

—>Andy

---

From: Tom Haynes <[email-masked]>
Date: Wed, 4 Mar 2015 17:44:21 -0800

> On Mar 4, 2015, at 11:44 AM, Adamson, Andy <[email-masked]> wrote:
> 
>> 

Please note that the quote you provide is a “may” and not a MUST.

The server hands out stateids and not the client. Thus the server
can hand out stateids that are unique across all clients.

From 5661:

   Stateid:  A stateid is a 128-bit quantity returned by a server that
      uniquely defines the open and locking states provided by the
      server for a specific open-owner or lock-owner/open-owner pair for
      a specific file and type of lock.

And

8.2.  Stateid Definition

   When the server grants a lock of any type (including opens, byte-
   range locks, delegations, and layouts), it responds with a unique
   stateid that represents a set of locks (often a single lock) for the
   same file, of the same type, and sharing the same ownership
   characteristics.  Thus, opens of the same file by different open-
   owners each have an identifying stateid.  Similarly, each set of
   byte-range locks on a file owned by a specific lock-owner has its own
   identifying stateid.  Delegations and layouts also have associated
   stateids by which they may be referenced.  The stateid is used as a
   shorthand reference to a lock or set of locks, and given a stateid,
   the server can determine the associated state-owner or state-owners
   (in the case of an open-owner/lock-owner pair) and the associated
   filehandle.  When stateids are used, the current filehandle must be
   the one associated with that stateid.

   All stateids associated with a given client ID are associated with a
   common lease that represents the claim of those stateids and the
   objects they represent to be maintained by the server.  See
   Section 8.3 for a discussion of the lease.

   The server may assign stateids independently for different clients.

^^ That is a may, not a MUST.

^^ Again, that is a may, not a MUST

   entirely different set of locks for a different client.  The stateid
   is always interpreted with respect to the client ID associated with
   the current session.  Stateids apply to all sessions associated with
   the given client ID, and the client may use a stateid obtained from
   one session on another session associated with the same client ID.

Nothing here states that the server cannot share stateids across
all clients.

> So it’s not a stateid! Call it something else.

It is a stateid. I do not see a need to re-invent the wheel and to introduce
yet another way of stating that this magic number uniquely identifies
a particular piece of state on the server.

Olga has successfully argued that by adding the stateid, it will make
implementation easier - note that I don’t say impossible, it reduces
complexity. I’m more than willing to add this copy stateid at this
late point.

---

From: "Adamson, Andy" <[email-masked]>
Date: Thu, 5 Mar 2015 14:15:30 +0000

> On Mar 4, 2015, at 8:44 PM, Tom Haynes <[email-masked]> wrote:
> 
>> 

Correct, and so a stateid can not be assumec to be unique.

> Nothing here states that the server cannot share stateids across

Everything says that a stateid can not be treated as unique.

> a particular piece of state on the server.

Adding a special stateid that is unique across all clientIDs  has the dis-advantage that current implementations that do re-use stateid bit patterns now need to change their basic stateid code. For what?? AFAICS there is no reason to call this a stateid.

Re-invent the wheel??? You seem perfectly willing to add a random number in "Section 4.10.1.2.  Inter-Server Copy via ONC RPC without RPCSEC_GSS” to identify state on the server!

Why not just use a random number and not incur the cost to those implementations?

> complexity. I’m more than willing to add this copy stateid at this

I can tell you are going with the cnr_stateid.

" The cnr_stateid is a copy stateid which uniquely describes the
 state needed on the source server to track the proposed copy.
 While normal stateids are only gaurenteed to be unique per client
 ID, the cnr_stateid is unique across all client IDs.”

This should be a MUST: "the cnr_stateid MUST be unique across all client IDs”

Perhaps I missed seeing the following as part of the cnr_stateid proposal:

Since we now have COPY_NOTIFY returning a cnr_stateid to uniquely describe the state needed on the source server to track the proposed copy, let’s go all the way and send the cnr_stateid to the destination server in the COPY operation, so that it can be used as the stateid for the READ by the destination server to the source server when the copy protocol is NFSv4.x. This will uniquely identify the READ as part of the copy, whereas presenting the ca_src_stateid as is now suggested does not.

If we do add cnr_stateid to COPY and use if for the READ we can get rid of the random number in the nfs URL in section 4.10.1.2 as the presentation of the cnr_stateid authenticates the channel - and does even better as it authenticates each READ. The suggested COMPOUND in section 4.10.1.2 is also not needed.

—>Andy

---

From: "Adamson, Andy" <[email-masked]>
Date: Thu, 5 Mar 2015 14:22:22 +0000

> On Mar 5, 2015, at 9:15 AM, Adamson, Andy <[email-masked]> wrote:
> 
> 

Indeed I did:

4.7.2.  Using NFSv4.x as the Copy Protocol

   The destination server MAY use standard NFSv4.x (where x >= 1)
   operations to read the data from the source server.  If NFSv4.x is
   used for the server-to-server copy protocol, the destination server
   can use the source filehandle and ca_src_stateid provided in the COPY
   request with standard NFSv4.x operations to read data from the source
   server.  Note that the ca_src_stateid MUST be the cnr_stateid
   returned from the source via the COPY_NOTIFY.

—>Andy

---

From: Tom Haynes <[email-masked]>
Date: Thu, 5 Mar 2015 08:54:00 -0800

> On Mar 5, 2015, at 6:15 AM, Adamson, Andy <[email-masked]> wrote:
> 
>> 

That is all an implementation detail. Your server is perfectly free to
create unique stateids for everything across all clients.

/*
 * Stateid
 */
struct stateid4 {
        uint32_t        seqid;
        opaque          other[12];
};

There are 12 bytes there to make it unique.

> Re-invent the wheel??? You seem perfectly willing to add a random number in "Section 4.10.1.2.  Inter-Server Copy via ONC RPC without RPCSEC_GSS” to identify state on the server!

No, I wasn’t - go check the archives - I was vehemently opposed and overridden by the WG.

> Why not just use a random number and not incur the cost to those implementations?

Olga’s problem statement was that there was a piece of state on the server that
needed to be tracked. It could be done with refcounts, etc. But it would be
easier to uniquely identify it.

The tracking of state is the job of a stateid.

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 5 Mar 2015 12:05:21 -0500

On Thu, Mar 5, 2015 at 11:54 AM, Tom Haynes
<[email-masked]> wrote:
> so a stateid is not unique global identifier for copy state on the server,
> as the same stateid bit pattern can exist under different clientids.
>
>
>
> Adding a special stateid that is unique across all clientIDs  has the
> dis-advantage that current implementations that do re-use stateid bit
> patterns now need to change their basic stateid code. For what?? AFAICS
> there is no reason to call this a stateid.
>
> Re-invent the wheel??? You seem perfectly willing to add a random number in
> "Section 4.10.1.2.  Inter-Server Copy via ONC RPC without RPCSEC_GSS” to
> identify state on the server!
>
>

Yes, specifying that stateid has to be unique does solve the problem
of matching COPY to the corresponding READ. However, I would have
preferred a simple random number instead of overloading the concept of
stateid. But, I will not fight for it.

> Since we now have COPY_NOTIFY returning a cnr_stateid to uniquely describe
> the state needed on the source server to track the proposed copy, let’s go
> all the way and send the cnr_stateid to the destination server in the COPY
> operation, so that it can be used as the stateid for the READ by the
> destination server to the source server when the copy protocol is NFSv4.x.
> This will uniquely identify the READ as part of the copy, whereas presenting
> the ca_src_stateid as is now suggested does not.
>
> If we do add cnr_stateid to COPY and use if for the READ we can get rid of
> the random number in the nfs URL in section 4.10.1.2 as the presentation of
> the cnr_stateid authenticates the channel - and does even better as it
> authenticates each READ. The suggested COMPOUND in section 4.10.1.2 is also
> not needed.
>
> —>Andy
>
>
>
>
>
> —>Andy
>
>
>
> As 4.2 inter-server copy stands now, the READ from the destination server
> (acting as an NFS cllient) to the source server sends the ca_src_stateid
> from the COPY (the OPEN sateid from the client) but the source server has no
> way to tell which clientid the stateid is associated with. Thus the source
> server can not tell the difference between two copies from different clients
> from the same source server to the same destination server where the
> stateid’s from each of the two client OPENs have the same bit pattern.
>
> —>Andy
>
>
> The information contained in this message and in any attachments is
> privileged and confidential, and may contain personal data. It is intended
> only for the use of the addressee. You should not read, copy, use, or
> disclose this information without authorisation. If you are not the intended
> recipient of this message, please e-mail the sender immediately and delete
> this message. Any unauthorised dissemination, disclosure, copying or use of
> the contents of this message is strictly prohibited and may result in legal
> action, including civil or criminal proceedings.
>
> NetApp-Disclaimer-v1.1
>
>
>