Subject: Re: [nfsv4] SERVERCOPY Procedure Proposal

From: Mitchell Erblich <[email-masked]>
Date: Wed, 1 Apr 2009 09:55:41 -0700

Group,

		Realizing that 1) my comments may be off track and
				       2) comments on the stated SERVERCOPY may be for  the  
incorrect doc..

		a) Shouldn't an arch doc specify whether holes will be filled.
		   - thus bytes written are actual byte count when holes are filled,
		   - or sum of file blocks that had dirty byes (wrt sparse files)  
equals count

		b) shouldn't status specify REMAINING lease time when one is  
specified,
		 c)    support infinite lease time (do it when you can in the  
background)

		d) IF failure is DST file to big, how many bytes were written vs  
wanted/ca_count
		    (support a short count and/or how far did we get when failed)

		e)  copy entire file (byte count = ca_count element of struct  
COPY4args)
		f)  failure return  - incorrect src file type

		Mitchell Erblich
		================================

On Mar 16, 2009, at 7:42 AM, James Lentini wrote:

>
> On Sat, 14 Mar 2009, Jim Callahan wrote:
>
>> I would like to propose a new NFS server procedure called SERVERCOPY
>> which performs an atomic file copy on an NFS server without routing  
>> data
>> through a client.   It would have the same arguments and return  
>> type as
>> the existing RENAME procedure except that it would only operate on
>> regular files.  Instead of renaming a file on the server it would
>> instruct the server to make a new target file which is an identical  
>> copy
>> of an existing source file.
>>
>> This would be equivalent in result to running the Unix "cp" command  
>> on
>> an NFS client to copy a file from one location to another on a single
>> server via the same NFS mount.   The advantage is that instead of
>> requiring the data to be send from the server to the client via many
>> READ procedures and then back to the server via many WRITE  
>> procedures,
>> this new SERVERCOPY procedure would be performed internally by the  
>> NFS
>> server.   No file data would need to be routed through the NFS  
>> client at
>> all.   This should lead to dramatically increased performance for  
>> file
>> copying when no reading or manipulation of the data involved is  
>> required
>> by the client.
>>
>> NFS server vendors would be free to implement this procedure in the  
>> most
>> optimal way for their particular software/hardware architecture.  For
>> single host NFS servers, this would avoid sending any file data  
>> over the
>> network.  For clustered implementations, there may still be some  
>> network
>> transmission involved but it still would be offloaded from the NFS
>> client and likely could be better optimized.
>>
>> Note that this would not provide any means for copying files between
>> different NFS servers or across NFS mounts.  It is entirely an
>> optimization for copying data internal to a single NFS server.
>>
>> -- 
>> Jim Callahan - President - Temerity Software <www.temerity.us>
>
> Jim,
>
> My colleagues and I have a proposal for a server-side copy procedure.
> We will be releasing this proposal after the March IETF meeting (and
> new drafts are eligible for submission). Our approach is along the
> lines you describe below. The operations provide a mechanism for
> copying a file from one location to another on a single NFS server.
>
> -james
>
> --
> James Lentini | NetApp | 781-768-5359 | [email-masked]
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 1 Apr 2009 13:19:20 -0400

>    - or sum of file blocks that had dirty byes (wrt 
>      sparse files) equals count

Why should the arch doc specify this?, i.e. what is the benefit of doing
so?

Since the user can't tell whether he is reading holes or zeros, what is
the benefit of specifying this behavior?

The downside is that it constrains implementation options.  You
certainly don't want to force a petabyte-sized hole to be filled in but
you really can't require that all holes be maintain since the source and
destination filesystems can have different block sizes.  If my source fs
has a 4K hole, it can't be maintained as a hole on a copy to a
destination with an 8K block size.

-----Original Message-----
From: Mitchell Erblich [mailto:[email-masked]] 
Sent: Wednesday, April 01, 2009 12:56 PM
To: Lentini, James
Cc: NFSv4 Working Group
Subject: Re: [nfsv4] SERVERCOPY Procedure Proposal

Group,

		Realizing that 1) my comments may be off track and
				       2) comments on the stated
SERVERCOPY may be for  the  
incorrect doc..

		a) Shouldn't an arch doc specify whether holes will be
filled.
		   - thus bytes written are actual byte count when holes
are filled,
		   - or sum of file blocks that had dirty byes (wrt
sparse files)  
equals count

		b) shouldn't status specify REMAINING lease time when
one is  
specified,
		 c)    support infinite lease time (do it when you can
in the  
background)

		d) IF failure is DST file to big, how many bytes were
written vs  
wanted/ca_count
		    (support a short count and/or how far did we get
when failed)

		e)  copy entire file (byte count = ca_count element of
struct  
COPY4args)
		f)  failure return  - incorrect src file type

		Mitchell Erblich

---

From: James Lentini <[email-masked]>
Date: Wed, 1 Apr 2009 14:34:20 -0400 (EDT)

On Wed, 1 Apr 2009, Mitchell Erblich wrote:

> 				       2) comments on the stated SERVERCOPY may be for  the incorrect doc..
> 
> 		   - or sum of file blocks that had dirty byes (wrt sparse files) equals count

I saw an answer from Dave on this.

> 		b) shouldn't status specify REMAINING lease time when one is specified,

Which status?

> 		 c)    support infinite lease time (do it when you can in the background)

A server could indicate an infinite lease by setting cnr_lease_time in 
COPY_NOTIFY4res to zero. That can be added in the next version.

> 		d) IF failure is DST file to big, how many bytes were written vs wanted/ca_count

This is provided for both synchronous and asynchronous copies. See 
COPY4res for the synchronous case and CB_COPY4args for the 
asynchronous case.

> 		e)  copy entire file (byte count = ca_count element of struct COPY4args)

This is supported. See the description of ca_count:

   A count of 0 (zero) requests that all bytes from ca_src_offset 
   through EOF be copied to the destination.

> 		f)  failure return  - incorrect src file type

The failure section is not exhaustive at this point. We have some of 
these cases identified (source is a directory see COPY's NFS4ERR_ISDIR 
return), but not all errors are explicitly listed.

> > > which performs an atomic file copy on an NFS server without routing data
> > > through a client.   It would have the same arguments and return type as
> > > instruct the server to make a new target file which is an identical copy
> > > This would be equivalent in result to running the Unix "cp" command on
> > > READ procedures and then back to the server via many WRITE procedures,
> > > this new SERVERCOPY procedure would be performed internally by the NFS
> > > server.   No file data would need to be routed through the NFS client at
> > > all.   This should lead to dramatically increased performance for file
> > > copying when no reading or manipulation of the data involved is required
> > > by the client.
> > > 
> > > NFS server vendors would be free to implement this procedure in the most
> > > single host NFS servers, this would avoid sending any file data over the
> > > network.  For clustered implementations, there may still be some network

---

From: Mitchell Erblich <[email-masked]>
Date: Thu, 2 Apr 2009 01:02:01 -0700

Comments inline...

Mitchell Erblich
==================

> Why should the arch doc specify this?, i.e. what is the benefit of  

	You must be pulling my leg..

	COW wrt procs is an acceptable practice and should be applied to files.

	However, if the orig file is a sparse file and later writes are to be  
applied, the copier
	may wish to know that as long as the file size does not change,  
writes WILL not
	fail due to ENOSPACE. This can only happen if a "expanding copy" is  
really wanted.
	
> the benefit of specifying this behavior?

	I am not sure that the user needs to be explicitly aware of sparcity  
of a file to
	be copied. The fs should be implicitly aware by the product of
	the number of fs blocks used by the file ands the number of fs blocks  
(in the
	simple case) the percentage of sparseness vs the specified size.

	However, it would be nice if the user knows that it is a sparse file  
object
	but specifies that the whole file object be copied, and the size is  
greater
	than the remaining space with the fs, that the return will be OK.

	Whether the user has the ability to query this information is probably
	fs implementation dependent. However, he should be able to pass a hint
	as to his intention of the copied file. Whether the fs can act upon  
this
	hint is also implementation dependent. But the hint should be able to  
be
	specified because of the obvious benefits of attempting to preserve
	the sparcity of the fs object.

	Removing the network from the fs data's path, and applying the copy to
	a server where the src and dst can reside local to the same server,  
adds
	the ability , IMO, to preserve the orig sparcity of the file and the  
ability
	to suggest the hint.

> destination with an 8K block size.

	Many years ago in this same galaxy (Star Wars), a well known company
	supported 4k in core / memory blocks and used a default 8k fs block
	and thus moved a min of two pages of data to the fs at a time. And
	holes were supported. fs was known as UFS.

	I am not suggesting a REQUIREMENT to support that the copy is
	a identifcal sparse file, just a hint as to whether to attempt to keep
	the level of sparcity or the wantomness to expand it.

	And yes, in this same galaxy a proposal was done to convert
	the fragment functionality within UFS, to support sparse frags
	and a variable fs block.

	And I don't understand if the 4k hole is at one end of the 8k
	fs block, why the 8k fs block then can not begin where the
	data is actually present. Your example, I think also assumes
	a fixed size on disk fs block. However, if there are certain offset
	restrictions, then those blocks will generate different internal
	fragmentation than the orig file blocks.

---

From: anshul <[email-masked]>
Date: Thu, 02 Apr 2009 14:25:24 +0530

> Mitchell Erblich

The NFS Server side copy Draft already has a COPY4_SPACE_RESERVED flag
defined for the COPY operation. When this flag is set in a COPY
operation, the Server would only perform the COPY if it can guarantee
that all writes to the copied file will not fail due to insufficient
space i.e. not return ENOSPACE. It is up to the server implementation as
to how it can guarantee that. So as Dave notes, we need not specify this
as it constrains implementation options.

Regards,
Anshul

---

From: Mitchell Erblich <[email-masked]>
Date: Thu, 2 Apr 2009 02:38:43 -0700

Inline..

	Mitchell Erblich
=====================

> as it constrains implementation options.

The "later writes" applies to "WRITES AFTER the sparce file is copied
without holes"  AND NOT specifying any form of atomicity.

It was only to be an EXAMPLE of why some user may NOT
want to specify the hint to keep the NEW file sparce,

In other words..
The user wants non extending read/modify/writes  at well known
locations in the NOW non sparce file NOT to be allocating any new
disk blocks.,

THIS is AFTER the SERVER COPY..

---

From: anshul <[email-masked]>
Date: Thu, 02 Apr 2009 15:18:26 +0530

> THIS is AFTER the SERVER COPY..

COPY4_SPACE_RESERVED flag is also set to specify the Server behaviour
AFTER the Server copy. If this flag is not set, then the Server that
supports COW is free to follow COW. If this flag is set, then the Server
implementation decides how it ensure that further writes will not fail
due to insufficient space.

Regards,
Anshul

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 2 Apr 2009 09:07:29 -0400

inline. 

-----Original Message-----
From: Mitchell Erblich [mailto:[email-masked]] 
Sent: Thursday, April 02, 2009 4:02 AM
To: Noveck, Dave
Cc: Lentini, James; NFSv4 Working Group
Subject: Re: [nfsv4] SERVERCOPY Procedure Proposal

> > Why should the arch doc specify this?, i.e. what is the benefit of 
> > doing so?
>

Somebody's pulling somebody's leg.  I'm kind of baffled by this 
discussion.

>	COW wrt procs is an acceptable practice and should be applied to
files.

It's an acceptable practice and whether it should be applied to files
or not is an implementation choice.

>	However, if the orig file is a sparse file and later writes are
to be applied, the copier
writes WILL not
>	fail due to ENOSPACE. This can only happen if a "expanding copy"
is really wanted.

"wanted" by whom?  This protocol extension is basically from the point
of view of the client.  In NFS, as of now, the client is not really 
aware of holes and the spec treats them as a piece of implementation
that
is hidden from the client.  As long as they look like zeros, that's good
enough.  Server implementations are free to choose whether and how they
will use holes.
	
> >
> > Since the user can't tell whether he is reading holes or zeros, what

> > is the benefit of specifying this behavior?
> >

>	I am not sure that the user needs to be explicitly aware of
sparcity of a file to
>	the number of fs blocks used by the file ands the number of fs
blocks (in the
>	simple case) the percentage of sparseness vs the specified size.

If the "architecture" specifies whether sparcity is to be maintained,
then 
the architecture has to be aware of sparcity.  As of today, the NFS 
architecture is not aware of the sparcity of files.  The model is that
you return the proper bytes including sets of zero and how you choose
to do that is up to you.  If your file system has a block of zeros and
mine doesn't, then that's OK.  If somebody writes a particular block of
data in five different files, and your fs has five copies and mine has
one shared, then that's OK too.  If NFS has to start being aware of
these sorts of choices other than in the sense of "do what you want as
long as you conform to the protocol", then every sort of implementation
choice such as the ones above becomes a protocol specification matter
and
most members of the working group would prefer not to have that
situation.

>	However, it would be nice if the user knows that it is a sparse
file object
>	but specifies that the whole file object be copied, and the size
is greater
>	than the remaining space with the fs, that the return will be
OK.

Instead of the user second-guessing the file system, he can just ask by
making
the request.  The server should return an error if it not possible and
the
server has the implementation information to know whether it can be
done.
It's best if the client doesn't need to know server filesystem
implementation 
data.

>	fs implementation dependent.

On a local fs it is, but in NFS he can't do it and I don't think he
should be
able to.
 
>     However, he should be able to pass a hint
>	as to his intention of the copied file. Whether the fs can act
upon this
>	hint is also implementation dependent. But the hint should be
able to be
>	the sparcity of the fs object.

I'm not against a hint a priori.

But for a hint to be worthwhile, there has to be some reason for
alternate
values to be specified.  I understand why a client would prefer
sparseness 
(or deduplication) to be maintained.  Why would the client (for some
particular
copies) prefer to not maintain that?  I understand that the server might
for 
various reasons and it has the implementation knowledge to make those
choices.

>	a server where the src and dst can reside local to the same
server, adds
>	the ability , IMO, to preserve the orig sparcity of the file and
the ability
>	to suggest the hint.

It adds the ability to maintain the sparseness where the source and
destination fs's allow it.  The ability to suggest the hint depends on
the working group being convinced that there is some value there.

> > certainly don't want to force a petabyte-sized hole to be filled in 
> > but you really can't require that all holes be maintain since the 
> > source and destination filesystems can have different block sizes.
If 
> > my source fs has a 4K hole, it can't be maintained as a hole on a
copy 
> > to a destination with an 8K block size.

>	holes were supported. fs was known as UFS.

It supported 8K holes that were 8K aligned.  That the buffer cache was
4K
is irrelevant.

Still if you had a 4K hole somewhere else, it could not be maintained if
the file were transferred to UFS.

>	the level of sparcity or the wantomness to expand it.

OK.  Now I understand what you are asking for.

The question I still have is why the client might need to specify two
different values of this hint.  What does it know that the server 
doesn't?

>	the fragment functionality within UFS, to support sparse frags

And if that proposal had been implemented (call the result UFS-- since
it uses less space), then many holes in a UFS-- file being copied to
a UFS fs would go away.  And the client really would not want to have
to deal with those issues.  

>	a fixed size on disk fs block. 

It's based on my experience with a range of file systems.  I'm aware
that there are other implementation choices and proposals but that 
sort of implementation is very common.  It's possible that file system
technology has taken a wrong turn somewhere and that we will recognize
this and move in a different direction.  But my guess is that we won't.

>     However, if there are certain offset

Yes.

-----------------------------------------------

filled.
are filled,
sparse files) 
one is 
written vs 
when failed)
struct

>>> copy of an existing source file.
>>>
>>> on an NFS client to copy a file from one location to another on a 
>>> single

>>> procedures, this new SERVERCOPY procedure would be performed 
>>> internally by the NFS
>>> required by the client.
>>> single host NFS servers, this would avoid sending any file data over

>>> the network.  For clustered implementations, there may still be some

>>> network transmission involved but it still would be offloaded from 
>>> the NFS client and likely could be better optimized.
>>>

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 2 Apr 2009 09:48:50 -0500

On Thu, Apr 02, 2009 at 01:02:01AM -0700, Mitchell Erblich wrote:
> >doing

["this" -> what to do about holes in files]

> 	to be  applied, the copier may wish to know that as long as the
> 	file size does not change,  writes WILL not fail due to
> 	ENOSPACE. This can only happen if a "expanding copy" is  really
> 	wanted.

+1

> >is the benefit of specifying this behavior?

The user most definitely can tell whether they are reading holes or
zeros.  Solaris has a public API for doing that.  If NFSv4.x can't
expose holes to clients, then NFSv4 needs to be fixed.  (Indeed, NFSv4.0
does not expose hole information to clients, and v4.1, I think, does not
either.)

Nico
--