Subject: [nfsv4] Proposal for a new SPACEMGMT operation

From: Christoph Hellwig <[email-masked]>
Date: Thu, 7 Nov 2013 21:26:08 +0100

1) Rationale

The NFSv4.2 drafts try to support hole punching and preallocation, but the
proposed WRITE_PLUS call is too far from the semantics a traditional NFS
client would expect to be be useful.  This document will define a new
SPACEMGMT operation than can better handle these requirements.  As a result
of an addition of this operation redundant functionality can be removed from
the WRITE_PLUS definition, but details of this as well as integration into
the overall draft RFC will be attempted later.

2) The mismatch between what WRITE_PLUS provides and what Linux expects

 a) Hole punch and preallocation semantics

    The Linux fallocate call must not overwrite data that is already
    written, but the WRITE_PLUS hole punching arm is defined as zeroing
    the whole region.  Checking the file size and adjusting the range
    on a client is an operation too racy to perform for common
    operations such as preallocation.

    The WRITE_PLUS call must not affect the space reservation, making
    it useless to punch an actual hole and release the space allocated
    to it.

    Besides the naming of the section the WRITE_PLUS hole punching arm
    thus does not perform actual hole punches or helps with preallocation
    but it just provides a way to offload zeroing of file data to the
    server.  While the latter is a useful feature by itself it should
    not be mixed up with space management functionality.
 
 b) Asynchronous nature

    Both a hole punch and a space preallocation should be quick operations
    and not require the offload state machine - any offloaded operation
    is most likely too slow to be useful for the users of the space
    management calls.

    For files systems that do support sparse files, hole punching is a
    generalization of the SETATTR operation that decreases the file
    size, and thus the same timing constraints apply.

    File systems not supporting sparse SHOULD NOT support hole punching
    as the hole punch is intended to reclaim space.

    Preallocation involves the allocation of space without writing data
    to it, either in form of physical blocks or file-specific
    reservation.  Supporting preallocation is a generalization of
    the PNFS_BLOCK_INVALID_DATA layout state in the pNFS block
    specification, and the same timing constraints as for a LAYOUTGET
    operation with a LAYOUTIOMODE4_RW iomode.

 b) Treatment as a data operation

    WRITE_PLUS is explicitly defined to be treated like writing zeros and
    integrates into the writes state machine, while hole punching and
    preallocation fundamentally are meta data operations.

3) Definition of the SPACEMGMT operation

The SPACEMGMT command is modeled after the IRIX/XFS space management fcntls
and the Linux fallocate system call.  Besides being heavily used application
interfaces themselves these interfaces are also to implement the Posix 1003.1
posix_fallocate() interface in glibc, and the to implement the T10 SBC
logical block provisioning interfaces in the Qemu hypervisor component.

Proposed RFC language below:

------------------------------------------------------------------------------
14.N.  Operation Z: SPACEMGMT
14.N.1.  ARGUMENT

   enum SPACEMGMT_type4 {
           SPM_PREALLOC                            = 1,
	   SPM_PUNCH_HOLE                          = 2,
   }

   struct SPACEMGMT4args {
           /* CURRENT_FH: file */
           stateid4        spm_stateid;
           offset4         spm_offset;
           length4         spm_length;
           bitmap4         spm_type;
   };

14.N.2.  RESULT

   struct SPACEMGMT4res {
           nfsstat4        status;
   };

14.N.3.  DESCRIPTION

   The SPACEMGMT operation offers manipulation of space allocated to a file.

   Whenever a client wishes to manipulate space allocation for blocks
   backing a particular region in the file, it calls the SPACEMGMT
   operation with the current filehandle set to the filehandle of the file
   in question, and the start offset and length in bytes of
   the region set in spm_offset and spm_length respectively.

   The SPACEMGMT operation will result in the space_used attribute being
   increased or decreased by the number of bytes manipulated.  The space_freed
   attribute may or may not change, depending on the support and whether the
   blocks backing the specified range were shared or not. 

   The SPACEMGMT operation is intended to expose actual space management
   functionality.  Servers SHOULD NOT emulate it using expensive zero
   writing or moving around of data, but return NFS4ERR_NOTSUPP if they do
   not support the specific operations.  Servers are not required to support
   each mode of the SPACEMGMT operation and can fail each one individually
   with NFS4ERR_NOTSUPP.

14.N.3.1.  Preallocation

   If spm_type is set to SPM_PREALLOC the server will ensure that backing
   blocks are allocated to the region specified by spm_offset and
   spm_length.  If the region contains already allocated blocks no
   action will be performed for those.  If the region lies partially or
   fully outside the current file size the file size will be set to
   spm_offset + spm_length implicitly, and if the previous file size
   was in the middle of a block the rest of the block will be zeroed.

   Blocks preallocated but not written to must return zeroes when read.

   Future writes to this region MUST NOT fail with NFSERR_NOSPC.

14.N.3.1.  Hole punching

   If spm_type is set to SPM_HOLE punch the server will deallocate all
   full blocks backing the file in the region specified by spm_offset and
   spm_length.

   Situations may arise where spm_offset and/or spm_offset + spm_length
   will not be aligned to a boundary that the server does allocations/
   deallocations in.  For most file systems, this is the block size of
   the file system.  In such a case, the server can manipulate as many
   bytes as it can in the region.  The blocks that cannot be manipulated
   MUST be zeroed.

   The server is not required to complete deallocating the blocks
   specified in the operation before returning, but it MUST guarantee
   that future reads from the region will return zeroes.
 
   The size attribute will remain unchanged.

   The operation will disable the the space reservation guarantee of the file.
------------------------------------------------------------------------------

4) Implementation

An implementation of the proposal will be provided after the initial dicussion
based on Anna Schumaker prototype for the Linux client and server.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 8 Nov 2013 15:54:36 +0000

>    as the hole punch is intended to reclaim space.

There is a functional requirement that you are missing, namely that this is not
just a way to save space in a filesystem, it is also a way to avoid needless
transfer of zeros across the wire.

I don't give a hoot as to whether or not we have WRITE_PLUS or a set of operators.

I do want us to maintain the ability to avoid the transfer of zeros. Whether that is
via SPACEMGMT_ASYNC or as a flag to SPACEMGMT, again, I don't care.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 8 Nov 2013 17:42:35 +0100

> transfer of zeros across the wire.

See my second mail for a replacement for that.  If we make the INITIALIZE
command (or whatever WRITE_PLUS arm it morphed to) actually write flat file
data it can replace that part, that's how things work in the block storage
block with WRITE SAME.

> I don't give a hoot as to whether or not we have WRITE_PLUS or a set of operators.

That's not my main motivation either.  While I think combining different
operations into a union is a bad idea unless they share semantics, the main
point is to get the semantics right.

---

From: "Noveck, David" <[email-masked]>
Date: Fri, 8 Nov 2013 11:59:27 -0500

> See my second mail for a replacement for that.  If we make the INITIALIZE command (or whatever WRITE_PLUS

> arm it  morphed to) actually write flat file data it can replace that part, that's how things work in the block storage

> block with WRITE SAME.

I don't think we can make file systems actually write file data.  They are always free to optimize this away.  The critical fact is that this is repetitive data, and not a hole.  It follows that it should look like repetitive data on READ_PLUS, although READ_PLUS should have a way to optimize the transfer.

> While I think combining different operations into a union is a bad idea unless they share semantics, the main point
> is to get the semantics right.

Agree but I think writing and writing data, whether repetitive or not, is semantically coherent.  So perhaps we need DATA and DATA_SAME arms for READ_PLUS and WRITE_PLUS.  For READ_PLUS a hole arm makes sense in READ_PLUS.  I guess that makes a HOLE arm for WRITE_PLUS, justified by parallelism, but don't feel strongly about that.

Note that doing this (hole creation) via {READ,WRITE}_PLUS means that hole creation is an operation that goes to the DS, which I'm OK with but you might not be.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christoph Hellwig
Sent: Friday, November 08, 2013 11:43 AM
To: Haynes, Tom
Cc: Christoph Hellwig; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Proposal for a new SPACEMGMT operation

On Fri, Nov 08, 2013 at 03:54:36PM +0000, Haynes, Tom wrote:

> There is a functional requirement that you are missing, namely that

> this is not just a way to save space in a filesystem, it is also a way

> to avoid needless transfer of zeros across the wire.

See my second mail for a replacement for that.  If we make the INITIALIZE command (or whatever WRITE_PLUS arm it morphed to) actually write flat file data it can replace that part, that's how things work in the block storage block with WRITE SAME.

> I don't give a hoot as to whether or not we have WRITE_PLUS or a set of operators.

That's not my main motivation either.  While I think combining different operations into a union is a bad idea unless they share semantics, the main point is to get the semantics right.

_______________________________________________

nfsv4 mailing list

[email-masked]<mailto:[email-masked]>

https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 8 Nov 2013 18:06:36 +0100

> I don't think we can make file systems actually write file data.  They are always free to optimize this away.  The critical fact is that this is repetitive data, and not a hole.  It follows that it…

Sorry for the imprecise wording - of course I didn't mean to imply they have to
write the flat data, but we have to make the _data model_ look like it's flat
data.  To take the block storage side as analogy again I know that many
disk will indeed write the pattern to blocks for a WRITE SAME, while many
higher end arrays will store a more compact representation.  The access mode on
is the same for both.

> Agree but I think writing and writing data, whether repetitive or not, is semantically coherent.  So perhaps we need DATA and DATA_SAME arms for READ_PLUS and WRITE_PLUS.  For READ_PLUS a hole arm…

The hole arm for READ_PLUS makes a lot of sense, and I want to keep it.  If
people strongly prefer to have an arm for WRITE_PLUS my proposal could easily
converted to that.

> Note that doing this (hole creation) via {READ,WRITE}_PLUS means that hole creation is an operation that goes to the DS, which I'm OK with but you might not be.

I have to say that the file layout is not someting I spent a lot of time
with, so considerations about it are better left to other people.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 8 Nov 2013 18:16:42 +0000

> people strongly prefer to have an arm for WRITE_PLUS my proposal could easily

I’m not married to WRITE_PLUS having arms or even existing.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 8 Nov 2013 19:34:11 +0100

> I’m not married to WRITE_PLUS having arms or even existing.

Let's the semantics and the the implementation choice using operations
or arms separate then.  Let's stick to the semantics in these two threads
here, and maybe we can have a discussion about unions on the write side
later.  I do think the union in READ_PLUS side is brilliant, btw - so no
need to argue about that from my side.