Subject: Re: [nfsv4] CLONE operation V2

From: Benny Halevy <[email-masked]>
Date: Thu, 02 Apr 2015 20:42:16 +0300

On 2015-03-31 12:15, Christoph Hellwig wrote:
> On Tue, Mar 31, 2015 at 07:07:34AM +0300, Benny Halevy wrote:
>> Hi Christoph, as I mentioned on the previous proposal I suggest to add
>> NFS4ERR_XDEV to the allowed errors for CLONE to be returned if SAVED_FH and
>> CURRENT_FH refer to files in different filesystems where the server cannot
>> clone blocks across filesystems.
> Sorry, I didn't manage to read that our of the garbage your mailer turned
> your text into.  This sounds reasonable, do you want to send a patch with
> the proposed wording?  Note that filesystem might not be the right term
> here as the clone scope might be very different on different systems.

No problem.  Sorry about the garbled html message.
Where is your git repo for the I-D?

The filesystem domain is a good point.  I agree that today's server
need a better term for allocation / backing-store block sharing domains.

Benny

---

From: Christoph Hellwig <[email-masked]>
Date: Thu, 2 Apr 2015 19:45:45 +0200

> Where is your git repo for the I-D?

It's Tom's repo: https://github.com/loghyr/NFSv4.2

> need a better term for allocation / backing-store block sharing domains.

Note that in NFS we always had the issue that a file system on the
server might not equal an export anyway, so similar wording to what 4.1
has for LINK might be fine:

   The statement "file and the target directory must reside within the
   same file system on the server" means that the fsid fields in the
   attributes for the objects are the same.  If they reside on different
   file systems, the error NFS4ERR_XDEV is returned.  This error may be
   returned by some servers when there is an internal partitioning of a
   file system that the LINK operation would violate.

---

From: Benny Halevy <[email-masked]>
Date: Fri, 03 Apr 2015 11:50:51 +0300

>     file system that the LINK operation would violate.

Requiring same fsid might be overly restrictive for some servers
that can share blocks across fsid boundaries.  But if folks are OK with
such a restriction it's somewhat easier to spec out.

---

From: Benny Halevy <[email-masked]>
Date: Fri, 03 Apr 2015 12:18:11 +0300

> such a restriction it's somewhat easier to spec out.

How about the following text:

diff --git a/nfsv42_middle_op_clone.xml b/nfsv42_middle_op_clone.xml
index f93e78d..7cd355f 100644
--- a/nfsv42_middle_op_clone.xml
+++ b/nfsv42_middle_op_clone.xml
@@ -35,6 +35,13 @@
      </t>

      <t>
+      If the server cannot share the data between the source file
+      and the destination file due to, e.g. file system or other
+      storage partition boundaries, the operation must fail and
+      return NFS4ERR_XDEV.
+    </t>
+
+    <t>
        Both SAVED_FH and CURRENT_FH must be regular files. If either
        SAVED_FH or CURRENT_FH are not regular files, the operation
        MUST fail and return NFS4ERR_WRONG_TYPE.

---

From: David Noveck <[email-masked]>
Date: Fri, 3 Apr 2015 08:53:26 -0400

While what you've written is clear to members of the working group,
there isn't enough background in the CLONE description to make it
understandable to the general spec audience or all IESG members.

Right now, all the spec says to set the stage for this is
"without actually copying the data, e.g., by using a copy on write
mechanism"

I'm not sure it's worth it to try to wordsmith a detailed explanation of
this.
I think we can just add NFS4ERR_XDEV to the list of errors CLONE
may validly return.  The definition of NFS4ERR_XDEV in RFC5661 is
sufficiently broad to tell clients what they need to know.

On Fri, Apr 3, 2015 at 5:18 AM, Benny Halevy <[email-masked]> wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 3 Apr 2015 11:25:30 -0700

> On Apr 3, 2015, at 5:53 AM, David Noveck <[email-masked]> wrote:
> 
> I'm not sure it's worth it to try to wordsmith a detailed explanation of this.

+1 and I will do this part while everyone argues if the rest is needed.

> 
> 
> On Fri, Apr 3, 2015 at 5:18 AM, Benny Halevy <[email-masked] <mailto:[email-masked]>> wrote:
> It's Tom's repo: https://github.com/loghyr/NFSv4.2 <https://github.com/loghyr/NFSv4.2>
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
>

---

From: Benny Halevy <[email-masked]>
Date: Sat, 04 Apr 2015 17:08:50 +0300

On 04/03/2015 09:25 PM, Thomas Haynes wrote:
>
>> On Apr 3, 2015, at 5:53 AM, David Noveck <[email-masked]
>> <mailto:[email-masked]>> wrote:
>>

Thanks.  This works for me too :)

Benny

>>             It's Tom's repo: https://github.com/loghyr/__NFSv4.2
>>             <https://github.com/loghyr/NFSv4.2>
>>
>>     _________________________________________________
>>     nfsv4 mailing list
>>     https://www.ietf.org/mailman/__listinfo/nfsv4
>>     <https://www.ietf.org/mailman/listinfo/nfsv4>
>>
>>

---

From: "J. Bruce Fields" <[email-masked]>
Date: Tue, 14 Apr 2015 17:23:52 -0400

I don't think clone_blksize is currently documented to be a
per-filesystem attribute, and I think it should be.

(Might be nice to have attribute scope documented in a standard way for
the various attributes, it's a little easy to overlook now.)

Also, support for cloning may vary depending on filesystem, so client
implementors could probably use a little help figuring out how to probe
for support.

So, maybe:

	- document a CLONE error for that case.  (I don't see an
	  existing error that's an obvious match.)

	- specify that the clone_blksize attribute is supported if and
	  only if the filesystem allows CLONE, so that clients can use
	  the supported_attrs attribute to query support.

--b.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 15 Apr 2015 09:23:14 +0300

On 2015-04-15 0:23, J. Bruce Fields
wrote:

Some servers may implement hybrid filesystems where different files
are

stored on different storage media.  Having just one per-filesystem
clone_blksize

attribute might force the server to expose the largest block size it
supports

across the filesystem so it might make sense to define the
per-filesystem

attribute as MUST implement when CLONE is supported for the
filesystem

and define an optional per-file clone_file_blksize attribute that
may override

the default one for the filesystem.

Agreed.

So, maybe:

That would also help for the per-file case.

Agreed.

I think we need both methods.

Benny

--b.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Wed, 15 Apr 2015 13:11:14 -0400

Benny, could you please fix your mailer?  It's really hard for some of
us to read your mail:

On Wed, Apr 15, 2015 at 09:23:14AM +0300, Benny Halevy wrote:
>     <blockquote cite="mid:[email-masked]"
>       type="cite">
>       <pre wrap="">I don't think clone_blksize is currently documented to be a
> per-filesystem attribute, and I think it should be.</pre>
>     </blockquote>
>     <br>
>     are<br>
>     clone_blksize<br>
>     supports<br>
>     per-filesystem<br>
>     filesystem<br>
>     may override<br>
>     the default one for the filesystem.<br>

Then we'd have to do that for every other per-filesystem attribute that
might actually vary between files.  That's not practical.  Instead I
think you just want to set the "homogeneous" attribute to false:

	http://tools.ietf.org/html/rfc5661#section-5.8.2.16

(I don't think the current linux client actually uses that, though.
Seems like it would be a pain to deal with.  In practice I think most
servers will want to be homogeneous.)

--b.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 16 Apr 2015 07:16:20 +0300

> Benny, could you please fix your mailer?  It's really hard for some of

Sigh, my mailer sends html formatted messages be default, my apologies.
This is fixed now, should be ok (as long as I remember to reply
from the right mailer :-/).

> 	http://tools.ietf.org/html/rfc5661#section-5.8.2.16

Yup, that would work.

> servers will want to be homogeneous.)

The pain is to revalidate per-filesystem attrs the client cares about,
per-file.  It'd be nice to have a mask of which perf-fs attrs are non-
homogeneous rather than (or in addition to) a bool value...

Benny

---

From: David Noveck <[email-masked]>
Date: Thu, 16 Apr 2015 06:25:57 -0400

The problem that this leaves you is that the description of clone that we
have makes no provision for clone when clone-blocksize of the two fules are
different.  It just says "clone block size", assuming that there is only
one.  It could by modified to say that the larger/GCM should be used and it
could say that XDEV should be returned if the two are different but the
issue does need to be addressed.

Are there any volunteers to propose a revision to 15.3 to address this?

---

From: Bruce Fields <[email-masked]>
Date: Thu, 16 Apr 2015 15:45:26 -0400

> issue does need to be addressed.

I don't know, is that case likely?  Might be a little harsh to make
every application always query blocksize on both files just in case some
server's really exotic.

Uh, I guess NFS4ERR_INVAL is the error for the case the arguments are
unaligned?

Then maybe:

	A server that supports CLONE may not support it on all exported
	filesystems.  It MUST support the clone_blksize attribute on any
	filesystem for which CLONE is supported, and SHOULD NOT support
	it on a filesystem for which CLONE is not supported.  Clients
	can use presence or absence of this attribute to determine CLONE
	support.

	A server that supports CLONE but not between the two files
	provided will return NFS4ERR_XDEV.

	If a server supports CLONE across filesystems, or between files
	on a filesystem for which the "homogeneous" attribute is false,
	then the server SHOULD arrange for the clone_blksize attribute
	to be the same on source and destination.

??

--b.

---

From: David Noveck <[email-masked]>
Date: Thu, 16 Apr 2015 21:45:42 -0400

> I don't know, is that case likely?

Actually it's exceedingly unlikely, and "exotic" as you say below but I
don't
think it is so weird that the spec can be written as if the issue does not
exist.

> Might be a little harsh to make

I don't see that anyone is talking about making the application (or the
client)
do anything like this.  Any requirements would be on the server in
executing the operation.  the client does not have to determine in advance
that the operation will succeed.

> Uh, I guess NFS4ERR_INVAL is the error for the case the arguments are

sound reasonable but the existing text doesn't say so.  It does mention a
number
of other reasons to return NFS4ERR_INVAL.

>        filesystems.

It would be clearer to say "might not support "

>        provided will return NFS4ERR_XDEV.

You need to adjust this to deal with a bunch of  issues here

   - I think we have to treat CLONE Support as something that may be
   different on different fs's.  As written, this says that if you do a clone
   and source and destination are on an fs that does not support cloning, you
   return NFS4ERR_XDEV.  i guess that's better than NFSERR$_NOTSUPP but I
   don't think NS4ERR_XDEV is right either.
   - There may be files that cannot be either source or destination of
   clone (because of something about those files.  In other words the issue
   might not be the relationship the two files but one of the file itself
   - Your text assumes that you cn do a clone wit h A a source and B as
   destination, the reverse id the case.  It mght not always be so.

 >       to be the same on source and destination.

The server can't arrange for this until he knows the source and
destination.  For example, in the homogeneous-false case, this is asking
the server to present the same clone_blocksize for each file in the fs or,
in other words to make the fs homogeneous.

Note that the complexity here would go away if we made CLONE copy the whole
file, which is what most people want to do anyway.  This would make source
and destination offsets zero, which makes most of the clone_blocksize issue
go away.

On Thu, Apr 16, 2015 at 3:45 PM, Bruce Fields <[email-masked]> wrote:

> > > >>       <pre wrap="">I don't think clone_blksize is currently
> documented
> > > to be a

---

From: Bruce Fields <[email-masked]>
Date: Fri, 17 Apr 2015 18:01:17 -0400

> > Might be a little harsh to make every application always query
> > blocksize on both files just in case some server's really exotic.
> 
On Thu, Apr 16, 2015 at 09:45:42PM -0400, David Noveck wrote:
> I don't see that anyone is talking about making the application (or
> the client) do anything like this.  Any requirements would be on the
> server in executing the operation.  the client does not have to
> determine in advance that the operation will succeed.

OK, nobody *has* to, but presumably somebody might *want* to, or we
wouldn't have bothered to provide the clone_blocksize attribute.

>    different on different fs's.  As written, this says that if you do
>    a clone and source and destination are on an fs that does not
>    support cloning, you return NFS4ERR_XDEV.  i guess that's better
>    than NFSERR$_NOTSUPP but I don't think NS4ERR_XDEV is right either.

Yeah, I was abusing XDEV to cover the unsupported case too--probably a
new NFS4ERR_CLONE_NOTSUPP would be clearer.  Or maybe something more
generic-sounding (NFS4ERR_NOTSUPP_ON_FS?) might turn out to be useful
for other ops some day.

> the server to present the same clone_blocksize for each file in the fs

Sure.  Or if it can't figure out that common clone_blocksize, it could
just turn off the cross-blocksize CLONE.  This shouldn't keep us up at
night.

> and destination offsets zero, which makes most of the clone_blocksize issue

I suppose I'd be OK with that, but if we're willing to drop support for
partial CLONEs entirely, why not just drop support for this one subcase
which even fewer people care about?

--b.

---

From: David Noveck <[email-masked]>
Date: Sat, 18 Apr 2015 09:20:28 -0400

---------- Forwarded message ----------
From: David Noveck <[email-masked]>
Date: Sat, Apr 18, 2015 at 7:47 AM
Subject: Re: [nfsv4] RFC: CLONE operation
To: Bruce Fields <[email-masked]>

> Or maybe something more

Sounds like a god idea.

> suppose I'd be OK with that, but if we're willing to drop support for

It's hard to be sure about the relative unimportance of these two cases.
The big
worry is that it could change.  As device technology changes, some of the
weird configurations that Benny is concerned about might come to seem less
weird.

I think there is a way to limit the cases affected to the intersection of
these two cases, each now unimportant/rare in itself.  how about adding the
 following after the third or fourth paragraph of 15.13.3?

Data areas to be cloned must be aligned according to the clone_blocksize
attribute of the source and destination files which generally need to be
the same   Errors in alignment result in NFS4ERR_INVAL.  In cases in

which the clone_blocksize is different for the two files, NFS4ERR_XDEV

will be returned.  An exception is the case in which the source and
destination offsets are both zero and the length of the area to be cloned
matches the length of the source file.  In this case, the clone_blocksize
is irrelevant to the success of the operation and NFS4ERR_XDEV need
not be returned simply because the clone_blocksize of the two files are
different.

On Fri, Apr 17, 2015 at 6:01 PM, Bruce Fields <[email-masked]> wrote:

> > > > have makes no provision for clone when clone-blocksize of the two
> fules
> > > are
> > > > one.  It could by modified to say that the larger/GCM should be used
> and
> > > it
> > > > > > Benny, could you please fix your mailer?  It's really hard for
> some
> > > of
> > > > > > us to read your mail:
> > > > >
> > > > > >>     Some servers may implement hybrid filesystems where
> different
> > > files
> > > > > >>     are<br>
> > > > > > Then we'd have to do that for every other per-filesystem
> attribute
> > > that

---

From: Benny Halevy <[email-masked]>
Date: Sun, 19 Apr 2015 11:14:23 +0300

On 2015-04-18 16:20, David Noveck wrote:
> From: *David Noveck* <[email-masked] <mailto:[email-masked]>>
> To: Bruce Fields <[email-masked] <mailto:[email-masked]>>
> 
> 
>> Or maybe something more
> It's hard to be sure about the relative unimportance of these two cases.   The big
> worry is that it could change.  As device technology changes, some of the weird configurations that Benny is concerned about might come to seem less weird.
> 
> I think there is a way to limit the cases affected to the intersection of these two cases, each now unimportant/rare in itself.  how about adding the  following after the third or fourth paragraph of 15.13.3?
> 
>     Data areas to be cloned must be aligned according to the clone_blocksize attribute of the source and destination files which generally need to be the same   Errors in alignment result in NFS4ERR_INVAL.  In cases in 
> 
>     will be returned.  An exception is the case in which the source and destination offsets are both zero and the length of the area to be cloned matches the length of the source file.  In this case, the clone_blocksize is irrelevant to the success of the operation and NFS4ERR_XDEV need not be returned simply because the clone_blocksize of the two files are different. 

We seem to be splitting hairs...

"generally need to be the same" doesn't help the implementer much
and different fs's have different restraints, some might not be able
to do byte-range clone altogether (e.g. when data are stored in cloud objects).

Also, the exception you mentioned could be problematic if the destination
file contains data beyond the source file's length and we're back to square
one regarding mixed block sizes.

I think we should leave more freedom to the implementations and provide
only general guidelines, without getting into too much details.

For example:

Data areas to be cloned must be aligned according to the clone_blksize
attribute of the source and destination files. Errors in alignment result
in NFS4ERR_INVAL.  In cases where the cloning of the requested byte range
is not allowed due to different source and destination clone_blksize or
other server storage constraints NFS4ERR_XDEV will result.

When the source and destination files reside on different filesystems
where the clone_blksize for the filesystems is different, The server SHOULD
allow to clone the whole source file to an empty, newly created destination
file, despite the different clock_blksize values.

Benny

---

From: David Noveck <[email-masked]>
Date: Sun, 19 Apr 2015 07:37:39 -0400

I'm OK with Benny's approach, except for one detail.  The last paragraph
should also cover the case of two files having different clone_blocksize
values on the same fs (when homogeneous is false).  Or we can drop the
case of fs's having inhomogenous clone_blocksize altogether.

On Sun, Apr 19, 2015 at 4:14 AM, Benny Halevy <[email-masked]> wrote:

> > From: *David Noveck* <[email-masked] <mailto:[email-masked]
> >>
> > worry is that it could change.  As device technology changes, some of
> the weird configurations that Benny is concerned about might come to seem
> less weird.
> >
> > I think there is a way to limit the cases affected to the intersection
> of these two cases, each now unimportant/rare in itself.  how about adding
> the  following after the third or fourth paragraph of 15.13.3?
> >
> >     Data areas to be cloned must be aligned according to the
> clone_blocksize attribute of the source and destination files which
> generally need to be the same   Errors in alignment result in
> NFS4ERR_INVAL.  In cases in
> >
> is irrelevant to the success of the operation and NFS4ERR_XDEV need not be
> returned simply because the clone_blocksize of the two files are different.
>
>

---

From: Benny Halevy <[email-masked]>
Date: Sun, 19 Apr 2015 19:27:20 +0300

On 2015-04-19 14:37, David Noveck wrote:
> I'm OK with Benny's approach, except for one detail.  The last paragraph should also cover the case of two files having different clone_blocksize values on the same fs (when homogeneous is false).

Good call.

> Or we can drop the

Though that not be the case in linux I think that commercial file servers
with different allocation block sizes are pretty common, e.g. 4 KB blocks
for small-ish files and fragments and 1 MB or more for larger files/chunks.
The client has no way of knowing that of course.

If dropping anything for simplicity sake is being considered, then I'd
be inclined to go for whole-file only and drop both clone_blksize and
support for byte range.

Benny

> 
> On Sun, Apr 19, 2015 at 4:14 AM, Benny Halevy <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     > From: *David Noveck* <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>     > To: Bruce Fields <[email-masked] <mailto:[email-masked]> <mailto:[email-masked] <mailto:[email-masked]>>>
>     >
>     >
>     >> Or maybe something more

---

From: Trond Myklebust <[email-masked]>
Date: Sun, 19 Apr 2015 13:29:22 -0400

On Sun, Apr 19, 2015 at 12:27 PM, Benny Halevy <[email-masked]> wrote:

> values on the same fs (when homogeneous is false).
>
> Good call.
>
> > Or we can drop the
I thought we were done with the use case discussion. Full file is not
sufficient for editing virtual machines.

> >     > From: *David Noveck* <[email-masked] <mailto:
> [email-masked]> <mailto:[email-masked] <mailto:
> [email-masked]>>>
> >     > To: Bruce Fields <[email-masked] <mailto:
> [email-masked]> <mailto:[email-masked] <mailto:
> [email-masked]>>>
> >     >
> >     >
> >     >> Or maybe something more
> >     > worry is that it could change.  As device technology changes, some
> of the weird configurations that Benny is concerned about might come to
> seem less weird.
> >     >
> >     > I think there is a way to limit the cases affected to the
> intersection of these two cases, each now unimportant/rare in itself.  how
> about adding the  following after the third or fourth paragraph of 15.13.3?
> >     >
> >     >     will be returned.  An exception is the case in which the
> source and destination offsets are both zero and the length of the area to
> be cloned matches the length of the source file.  In this case, the
> clone_blocksize is irrelevant to the success of the operation and
> NFS4ERR_XDEV need not be returned simply because the clone_blocksize of the
> two files are different.
> >
> >

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 20 Apr 2015 10:36:58 -0400

> sufficient for editing virtual machines.

I missed that, or forgot it.  Do you remember where it was discussed?

--b.