Subject: [nfsv4] [PATCH, RFC] revisit space reservations, introduce ALLOCATE operation

From: Christoph Hellwig <[email-masked]>
Date: Tue, 19 Nov 2013 18:08:13 +0100

There's been some mostly private feedback to my SPACEMGMGT proposal.

One that I heard a few times and agree is that it's not a good idea
to multiplex the preallocation and the hole punch call.  I tend to agree
with that, it's just something that grew out of the fallocate interface
I'm familiar with on Linux.

Second there's been suggestion that the WRITE_PLUS hole case still makes
sense as it's the counter part to the READ_SIZE, so many people would
like to retain it in a similar form.  I also have to admit that the
deallocating hole punch case in it is workable, even if the offload
requirement is annoying to the client.

Based on this I've come up with the ALLOCATE proposal below, which
intends to provide a useful backend for posix_fallocate(2), fallocate(2)
and also a replace the space_reserved attribute with a more general
implementation.

Compared to the WRITE_PLUS hole arm and SPACEMGMT the wording has changed
to not talk about allocating blocks but reserving space.  Given that
the client can't really see the actual block allocation behind the sense
I think this is a much better language, and it also allows us to converge
the way fallocate is used for preallocation with the way we do whole file
reservations for the hypervisor use case - in the end the server has
to decide how a reservation works.  For a traditional Unix-like like
filesystems it will perform actual preallocation using unwritten extents,
but a log-structured or COW filesystem might just do some sort of
a space reservation.  The space_reserved flag is dropped as it can be
trivially emulated by the client using the following scheme:

 - instead of extending the file size by WRITEs or SETATTRSs behind the
   file size ALLOCATE must be used
 - no deallocating WRITE_PLUS hole operation should be performed

note that the latter іs only a requirement if you want exactly the
same semantics, in general it might make sense to allow the hypervisor
to still deallocate blocks without doing a unset space_reserved,
hole punch, set space_reserve sequence.

I'd also like to mention that some hypervisors very much want the
fallocate semantics, e.g. KVM with the qcow2 format these days usually
preallocates space for it's own metadata, but not for actual file data
so that it can provide T10 logical block provisioning semantics.

Please find the proposal as a diff against the spec below.

diff --git a/Makefile b/Makefile
index 500dba3..24f65dd 100755
--- a/Makefile
+++ b/Makefile
@@ -276,6 +276,8 @@ AUTOGEN =	\
 		$(SPITGENXML) \
 		autogen/access_args.xml \
 		autogen/access_res.xml \
+		autogen/allocate_args.xml \
+		autogen/allocate_res.xml \
 		autogen/backchannel_ctl_args.xml \
 		autogen/backchannel_ctl_res.xml \
 		autogen/bind_conn_to_session_args.xml \
@@ -440,6 +442,7 @@ IDXMLSRC_BASE = \
         ${DOC_PREFIX}_middle_fileattributes.xml \
 	${DOC_PREFIX}_middle_op_mandlist.xml \
 	${DOC_PREFIX}_middle_op_aaa.xml \
+	${DOC_PREFIX}_middle_op_allocate.xml \
 	${DOC_PREFIX}_middle_op_copy.xml \
 	${DOC_PREFIX}_middle_op_offload_abort.xml \
 	${DOC_PREFIX}_middle_op_copy_notify.xml \
diff --git a/Makefile.beta b/Makefile.beta
index cedfaf7..480603b 100755
--- a/Makefile.beta
+++ b/Makefile.beta
@@ -270,6 +270,8 @@ AUTOGEN =	\
 		$(SPITGENXML) \
 		autogen/access_args.xml \
 		autogen/access_res.xml \
+		autogen/allocate_args.xml \
+		autogen/allocate_res.xml \
 		autogen/backchannel_ctl_args.xml \
 		autogen/backchannel_ctl_res.xml \
 		autogen/bind_conn_to_session_args.xml \
@@ -434,6 +436,7 @@ IDXMLSRC_BASE = \
         ${DOC_PREFIX}_middle_fileattributes.xml \
 	${DOC_PREFIX}_middle_op_mandlist.xml \
 	${DOC_PREFIX}_middle_op_aaa.xml \
+	${DOC_PREFIX}_middle_op_allocate.xml \
 	${DOC_PREFIX}_middle_op_copy.xml \
 	${DOC_PREFIX}_middle_op_offload_abort.xml \
 	${DOC_PREFIX}_middle_op_copy_notify.xml \
diff --git a/dotx.d/allocate_args.x b/dotx.d/allocate_args.x
new file mode 100644
index 0000000..b25b7fd
--- /dev/null
+++ b/dotx.d/allocate_args.x
@@ -0,0 +1,6 @@
+struct ALLOCATE4args {
+	/* CURRENT_FH: file */
+	stateid4        aa_stateid;
+        offset4         aa_offset;
+        length4         aa_length;
+};
diff --git a/dotx.d/allocate_res.x b/dotx.d/allocate_res.x
new file mode 100644
index 0000000..1c0e8df
--- /dev/null
+++ b/dotx.d/allocate_res.x
@@ -0,0 +1,3 @@
+struct ALLOCATE4res {
+	nfsstat4		ar_status;
+};
diff --git a/dotx.d/head.x b/dotx.d/head.x
index ce65cdb..6da0dfa 100644
--- a/dotx.d/head.x
+++ b/dotx.d/head.x
@@ -498,7 +498,6 @@ typedef change_policy4	fattr4_change_policy;
 /*
  * attributes new to NFSv4.2
  */
-typedef bool		fattr_space_reserved;
 typedef uint64_t	fattr_space_freed;
 typedef change_attr_type4
 		fattr4_change_attr_type;
diff --git a/dotx.d/spit_types.sh b/dotx.d/spit_types.sh
index 40a2468..008503b 100755
--- a/dotx.d/spit_types.sh
+++ b/dotx.d/spit_types.sh
@@ -1001,6 +1001,7 @@ enum nfs_opnum4 {
  OP_READ_PLUS		= 65,
  OP_SEEK		= 66,
  OP_IO_ADVISE		= 67,
+ OP_ALLOCATE		= 68,
  OP_ILLEGAL		= 10044
 };
 EOF
@@ -1121,6 +1122,7 @@ union nfs_argop4 switch (nfs_opnum4 argop) {
  case OP_READ_PLUS:	READ_PLUS4args opread_plus;
  case OP_SEEK:		SEEK4args opseek;
  case OP_IO_ADVISE:	IO_ADVISE4args opio_advise;
+ case OP_ALLOCATE:	ALLOCATE4args opallocate;
 
  /* Operations not new to NFSv4.1 */
  case OP_ILLEGAL:	void;
@@ -1251,6 +1253,7 @@ union nfs_resop4 switch (nfs_opnum4 resop) {
  case OP_READ_PLUS:	READ_PLUS4res opread_plus;
  case OP_SEEK:		SEEK4res opseek;
  case OP_IO_ADVISE:	IO_ADVISE4res opio_advise;
+ case OP_ALLOCATE:	ALLOCATE4res opallocate_plus;
 
  /* Operations not new to NFSv4.1 */
  case OP_ILLEGAL:	ILLEGAL4res opillegal;
diff --git a/nfsv42_middle_errors.xml b/nfsv42_middle_errors.xml
index f279f46..0756b2c 100644
--- a/nfsv42_middle_errors.xml
+++ b/nfsv42_middle_errors.xml
@@ -1335,6 +1335,41 @@
         NFS4ERR_TOO_MANY_OPS
       </c>
 
+      <c>ALLOCATE</c>
+      <c>
+        NFS4ERR_ACCESS,
+        NFS4ERR_ADMIN_REVOKED,
+        NFS4ERR_BADXDR,
+        NFS4ERR_BAD_STATEID,
+        NFS4ERR_DEADSESSION,
+        NFS4ERR_DELAY,
+        NFS4ERR_DELEG_REVOKED,
+        NFS4ERR_DQUOT,
+        NFS4ERR_EXPIRED,
+        NFS4ERR_FBIG,
+        NFS4ERR_FHEXPIRED,
+        NFS4ERR_GRACE,
+        NFS4ERR_INVAL,
+        NFS4ERR_IO,
+        NFS4ERR_ISDIR,
+        NFS4ERR_MOVED,
+        NFS4ERR_NOFILEHANDLE,
+        NFS4ERR_NOSPC,
+        NFS4ERR_OLD_STATEID,
+        NFS4ERR_OPENMODE,
+        NFS4ERR_OP_NOT_IN_SESSION,
+        NFS4ERR_REP_TOO_BIG,
+        NFS4ERR_REP_TOO_BIG_TO_CACHE,
+        NFS4ERR_REQ_TOO_BIG,
+        NFS4ERR_RETRY_UNCACHED_REP,
+        NFS4ERR_ROFS,
+        NFS4ERR_SERVERFAULT,
+        NFS4ERR_STALE,
+        NFS4ERR_SYMLINK,
+        NFS4ERR_TOO_MANY_OPS,
+        NFS4ERR_WRONG_TYPE
+      </c>
+
       <c>BACKCHANNEL_CTL</c>
       <c>
         NFS4ERR_BADXDR,
diff --git a/nfsv42_middle_fileattributes.xml b/nfsv42_middle_fileattributes.xml
index 3d6672f..b9ed982 100644
--- a/nfsv42_middle_fileattributes.xml
+++ b/nfsv42_middle_fileattributes.xml
@@ -56,7 +56,6 @@
       <c>change_attr_type</c> <c>79</c> <c>change_attr_type4</c>  <c>R  </c> <c><xref target="ss:fattr:chattr" /></c>
       <c>sec_label</c>        <c>80</c> <c>sec_label4</c>         <c>R W</c> <c><xref target="ss:fattr:sec" /></c>
       <c>change_sec_label</c> <c>81</c> <c>change_sec_label4</c>  <c>R  </c> <c><xref target="ss:fattr:chsec" /></c>
-      <c>space_reserved</c>   <c>77</c> <c>boolean</c>            <c>R W</c> <c><xref target="ss:fattr:sr" /></c>
       <c>space_freed</c>      <c>78</c> <c>length4</c>            <c>R  </c> <c><xref target="ss:fattr:sf" /></c>
     </texttable>
   </section>
@@ -214,51 +213,6 @@ typedef uint32_t  policy4;
       </t>
     </section>
 
-    <section toc='exclude' anchor="ss:fattr:sr" title="Attribute 77: space_reserved">
-      <t>
-        The space_reserve attribute is a read/write attribute of type
-        boolean.  It is a per file attribute and applies during the
-        lifetime of the file or until it is turned off.  When the space_reserved
-        attribute is set via SETATTR, the server must ensure that there is
-        disk space to accommodate every byte in the file before it can return
-        success.  If the server cannot guarantee this, it must return
-        NFS4ERR_NOSPC.
-      </t>
-
-      <t>
-        If the client tries to grow a file which has the space_reserved
-        attribute set, the server must guarantee that there is disk space to
-        accommodate every byte in the file with the new size before it can
-        return success.  If the server cannot guarantee this, it must return
-        NFS4ERR_NOSPC.
-      </t>
-
-      <t>
-        It is not required that the server allocate the space to the file
-        before returning success.  The allocation can be deferred, however,
-        it must be guaranteed that it will not fail for lack of space.
-      </t>
-
-      <t>
-        The value of space_reserved can be obtained at any time through
-        GETATTR. If the size is retrieved at the same time, the client
-        can determine the size of the reservation.
-      </t>
-
-      <t>
-        In order to avoid ambiguity, the space_reserve bit cannot be set
-        along with the size bit in SETATTR.  Increasing the size of a file
-        with space_reserve set will fail if space reservation cannot be
-        guaranteed for the new size.  If the file size is decreased, space
-        reservation is only guaranteed for the new size.
-        If a hole is punched into the file, then the reservation is not
-        changed.
-      </t>
-
-      <t>
-      </t>
-    </section>
-
     <section toc='exclude' anchor="ss:fattr:sf" title="Attribute 78: space_freed">
       <t>
         space_freed gives the number of bytes freed if the file is deleted.
diff --git a/nfsv42_middle_introduction.xml b/nfsv42_middle_introduction.xml
index 3e24bc6..3bdbf13 100644
--- a/nfsv42_middle_introduction.xml
+++ b/nfsv42_middle_introduction.xml
@@ -113,8 +113,8 @@
       <t>
         When a file is sparse, one concern applications have is ensuring
         that there will always be enough data blocks available for the
-        file during future writes. A new attribute, space_reserved
-        (see <xref target='ss:fattr:sr' />) provides the client
+        file during future writes. ALLOCATE
+	(see <xref target='op:allocate' />) allows a client to request
         a guarantee that space will be available.
       </t>
     </section>
diff --git a/nfsv42_middle_op_allocate.xml b/nfsv42_middle_op_allocate.xml
new file mode 100644
index 0000000..e116f74
--- /dev/null
+++ b/nfsv42_middle_op_allocate.xml
@@ -0,0 +1,39 @@
+<!-- Copyright (C) The IETF Trust (2013) -->
+<!-- Copyright (C) The Internet Society (2013) -->
+
+<section anchor='op:allocate' title='Operation 68: ALLOCATE'>
+  <section toc='exclude' title="ARGUMENT">
+    <?rfc include='autogen/allocate_args.xml'?>
+  </section>
+
+  <section toc='exclude' title="RESULT">
+    <?rfc include='autogen/allocate_res.xml'?>
+  </section>
+
+  <section toc='exclude' title="DESCRIPTION">
+    <t>
+      Whenever a client wishes to reserve space for a region in a file it calls
+      the ALLOCATE operation with the current filehandle set to the filehandle
+      of the file in question, and the start offset and length in bytes of
+      the region set in aa_offset and aa_length respectively.
+    </t>
+
+    <t>
+      The server will ensure that backing blocks are reserved to the region
+      specified by aa_offset and aa_length. If the region lies partially or
+      fully outside the current file size the file size will be set to
+      aa_offset + aa_length implicitly. Any bytes that are reserved but
+      do not contain data must return zero when read.
+    </t>
+
+    <t>
+      Future writes to this region MUST NOT fail with NFSERR_NOSPC.
+    </t>
+
+    <t>
+      The ALLOCATE operation will result in the space_used attribute and
+      space_freed attributes being increased by the number of bytes reserved
+      unless they were previously reserved or written and not shared.
+    </t>
+  </section>
+</section>
diff --git a/nfsv42_middle_op_write_plus.xml b/nfsv42_middle_op_write_plus.xml
index ea3f21f..85d223e 100644
--- a/nfsv42_middle_op_write_plus.xml
+++ b/nfsv42_middle_op_write_plus.xml
@@ -106,14 +106,9 @@
         deallocated.  The space_freed attribute
         may or may not decrease, depending on the support and whether the
         blocks backing the specified range were shared or not.  The size
-        attribute will remain unchanged.
-      </t>
-
-      <t>
-        The WRITE_PLUS operation MUST NOT change the space reservation
-        guarantee of the file.  While the server can deallocate the blocks
-        specified by di_offset and di_length, future writes to this region
-        MUST NOT fail with NFSERR_NOSPC.
+        attribute will remain unchanged.  If space was reserved for some
+        or all of the bytes that were converted to a hole that space
+        reservation will be dropped by WRITE_PLUS.
       </t>
     </section>
 
diff --git a/nfsv42_middle_space.xml b/nfsv42_middle_space.xml
index 44f0e40..3e2487e 100644
--- a/nfsv42_middle_space.xml
+++ b/nfsv42_middle_space.xml
@@ -4,21 +4,28 @@
 <section anchor="sec:space" title="Space Reservation">
   <section anchor="ss:space:intro" title="Introduction">
     <t>
-      Applications such as hypervisors want to be able to
-      reserve space for a file, report the amount of
-      actual disk space a file occupies, and free-up the backing space of a
-      file when it is not required.  In virtualized environments, virtual
-      disk files are often stored on NFS mounted volumes.  Since virtual
-      disk files represent the hard disks of virtual machines, hypervisors
-      often have to guarantee certain properties for the file.
+      Applications want to be able to reserve space for a file, report the
+      amount of actual disk space a file occupies, and free-up the backing
+      space of a file when it is not required.
     </t>
 
     <t>
-      One such example is space reservation.  When a hypervisor creates a
-      virtual disk file, it often tries to preallocate the space for the
-      file so that there are no future allocation related errors during the
-      operation of the virtual machine.  Such errors prevent a virtual
-      machine from continuing execution and result in downtime.
+      One example is the posix_fallocate(2) which allows applications to
+      ask for space reservations from the operating system, usually to
+      provide a better file layout and reduce overhead for random or
+      slow growing file appending workloads.
+    </t>
+
+    <t>
+      Another example is space reservation for virtual disks in a hypervisor.
+      In virtualized environments, virtual disk files are often stored on NFS
+      mounted volumes.  
+
+      When a hypervisor creates a virtual disk file, it often tries to
+      preallocate the space for the file so that there are no future
+      allocation related errors during the operation of the virtual machine.
+      Such errors prevent a virtual machine from continuing execution and
+      result in downtime.
     </t>
 
     <t>
@@ -32,13 +39,14 @@
     </t>
 
     <t>
-      We define a "reservation" as being the combination of the 
-      space_reserved attribute (see <xref target='ss:fattr:sr' />) and
-      the size attribute (see Section 5.8.1.5 of <xref target="RFC5661" />).
-      If space_reserved attribute is
-      set on a file, it is guaranteed that writes that do not grow the file
-      past the size will not fail with NFS4ERR_NOSPC. Once the size is
-      changed, then the reservation is changed to that new size.
+      The new ALLOCATE operation (see <xref target='op:allocate' />)
+      allows a client to request a guarantee that space will be available.
+      The ALLOCATE operation gurantees that any future writes to the
+      region it was successfully called for will not fail with NFS4ERR_NOSPC.
+
+      A reservations established by ALLOCATE will be given up on a successful
+      cal to the WRITE_PLUS (see <xref target='op:write_plus' />) that
+      deallocates blocks.
     </t>
 
     <t>
@@ -86,11 +94,6 @@
 
     <t>
       <list style="hanging">
-        <t hangText="space_reserved">
-          This attribute specifies that writes to the reserved
-          area of the file will not fail with NFS4ERR_NOSPACE.
-        </t>
-
         <t hangText="space_freed">
           This attribute specifies the space freed when a file is
           deleted, taking block sharing into consideration.

---

From: Marc Eshel <[email-masked]>
Date: Wed, 20 Nov 2013 09:00:03 -0800

I like the idea of having a separate operation for allocation.
Marc.

From:   Christoph Hellwig <[email-masked]>
To:     Tom Haynes <[email-masked]>, 
Cc:     [email-masked]
Date:   11/19/2013 09:08 AM
Subject:        [nfsv4] [PATCH, RFC] revisit space reservations, introduce 
ALLOCATE        operation
Sent by:        "nfsv4" <[email-masked]>

note that the latter ?s only a requirement if you want exactly the
same semantics, in general it might make sense to allow the hypervisor
to still deallocate blocks without doing a unset space_reserved,
hole punch, set space_reserve sequence.

 

+      <c>ALLOCATE</c>
+      <c>
+        NFS4ERR_ACCESS,
+        NFS4ERR_ADMIN_REVOKED,
+        NFS4ERR_BADXDR,
+        NFS4ERR_BAD_STATEID,
+        NFS4ERR_DEADSESSION,
+        NFS4ERR_DELAY,
+        NFS4ERR_DELEG_REVOKED,
+        NFS4ERR_DQUOT,
+        NFS4ERR_EXPIRED,
+        NFS4ERR_FBIG,
+        NFS4ERR_FHEXPIRED,
+        NFS4ERR_GRACE,
+        NFS4ERR_INVAL,
+        NFS4ERR_IO,
+        NFS4ERR_ISDIR,
+        NFS4ERR_MOVED,
+        NFS4ERR_NOFILEHANDLE,
+        NFS4ERR_NOSPC,
+        NFS4ERR_OLD_STATEID,
+        NFS4ERR_OPENMODE,
+        NFS4ERR_OP_NOT_IN_SESSION,
+        NFS4ERR_REP_TOO_BIG,
+        NFS4ERR_REP_TOO_BIG_TO_CACHE,
+        NFS4ERR_REQ_TOO_BIG,
+        NFS4ERR_RETRY_UNCACHED_REP,
+        NFS4ERR_ROFS,
+        NFS4ERR_SERVERFAULT,
+        NFS4ERR_STALE,
+        NFS4ERR_SYMLINK,
+        NFS4ERR_TOO_MANY_OPS,
+        NFS4ERR_WRONG_TYPE
+      </c>
+
       <c>BACKCHANNEL_CTL</c>
       <c>
         NFS4ERR_BADXDR,
diff --git a/nfsv42_middle_fileattributes.xml 
b/nfsv42_middle_fileattributes.xml
index 3d6672f..b9ed982 100644
--- a/nfsv42_middle_fileattributes.xml
+++ b/nfsv42_middle_fileattributes.xml
@@ -56,7 +56,6 @@
       <c>change_attr_type</c> <c>79</c> <c>change_attr_type4</c>  <c>R 
</c> <c><xref target="ss:fattr:chattr" /></c>
       <c>sec_label</c>        <c>80</c> <c>sec_label4</c>         <c>R 
W</c> <c><xref target="ss:fattr:sec" /></c>
       <c>change_sec_label</c> <c>81</c> <c>change_sec_label4</c>  <c>R 
</c> <c><xref target="ss:fattr:chsec" /></c>
-      <c>space_reserved</c>   <c>77</c> <c>boolean</c>            <c>R 
W</c> <c><xref target="ss:fattr:sr" /></c>
       <c>space_freed</c>      <c>78</c> <c>length4</c>            <c>R 
</c> <c><xref target="ss:fattr:sf" /></c>
     </texttable>
   </section>
@@ -214,51 +213,6 @@ typedef uint32_t  policy4;
       </t>
     </section>
 
-    <section toc='exclude' anchor="ss:fattr:sr" title="Attribute 77: 
space_reserved">
-      <t>
-        The space_reserve attribute is a read/write attribute of type
-        boolean.  It is a per file attribute and applies during the
-        lifetime of the file or until it is turned off.  When the 
space_reserved
-        attribute is set via SETATTR, the server must ensure that there 
is
-        disk space to accommodate every byte in the file before it can 
return
-        success.  If the server cannot guarantee this, it must return
-        NFS4ERR_NOSPC.
-      </t>
-
-      <t>
-        If the client tries to grow a file which has the space_reserved
-        attribute set, the server must guarantee that there is disk space 
to
-        accommodate every byte in the file with the new size before it 
can
-        return success.  If the server cannot guarantee this, it must 
return
-        NFS4ERR_NOSPC.
-      </t>
-
-      <t>
-        It is not required that the server allocate the space to the file
-        before returning success.  The allocation can be deferred, 
however,
-        it must be guaranteed that it will not fail for lack of space.
-      </t>
-
-      <t>
-        The value of space_reserved can be obtained at any time through
-        GETATTR. If the size is retrieved at the same time, the client
-        can determine the size of the reservation.
-      </t>
-
-      <t>
-        In order to avoid ambiguity, the space_reserve bit cannot be set
-        along with the size bit in SETATTR.  Increasing the size of a 
file
-        with space_reserve set will fail if space reservation cannot be
-        guaranteed for the new size.  If the file size is decreased, 
space
-        reservation is only guaranteed for the new size.
-        If a hole is punched into the file, then the reservation is not
-        changed.
-      </t>
-
-      <t>
-      </t>
-    </section>
-
     <section toc='exclude' anchor="ss:fattr:sf" title="Attribute 78: 
space_freed">
       <t>
         space_freed gives the number of bytes freed if the file is 
deleted.
diff --git a/nfsv42_middle_introduction.xml 
b/nfsv42_middle_introduction.xml
index 3e24bc6..3bdbf13 100644
--- a/nfsv42_middle_introduction.xml
+++ b/nfsv42_middle_introduction.xml
@@ -113,8 +113,8 @@
       <t>
         When a file is sparse, one concern applications have is ensuring
         that there will always be enough data blocks available for the
-        file during future writes. A new attribute, space_reserved
-        (see <xref target='ss:fattr:sr' />) provides the client
+        file during future writes. ALLOCATE
+                (see <xref target='op:allocate' />) allows a client to 
request
         a guarantee that space will be available.
       </t>
     </section>
diff --git a/nfsv42_middle_op_allocate.xml b/nfsv42_middle_op_allocate.xml
new file mode 100644
index 0000000..e116f74
--- /dev/null
+++ b/nfsv42_middle_op_allocate.xml
@@ -0,0 +1,39 @@
+<!-- Copyright (C) The IETF Trust (2013) -->
+<!-- Copyright (C) The Internet Society (2013) -->
+
+<section anchor='op:allocate' title='Operation 68: ALLOCATE'>
+  <section toc='exclude' title="ARGUMENT">
+    <?rfc include='autogen/allocate_args.xml'?>
+  </section>
+
+  <section toc='exclude' title="RESULT">
+    <?rfc include='autogen/allocate_res.xml'?>
+  </section>
+
+  <section toc='exclude' title="DESCRIPTION">
+    <t>
+      Whenever a client wishes to reserve space for a region in a file it 
calls
+      the ALLOCATE operation with the current filehandle set to the 
filehandle
+      of the file in question, and the start offset and length in bytes 
of
+      the region set in aa_offset and aa_length respectively.
+    </t>
+
+    <t>
+      The server will ensure that backing blocks are reserved to the 
region
+      specified by aa_offset and aa_length. If the region lies partially 
or
+      fully outside the current file size the file size will be set to
+      aa_offset + aa_length implicitly. Any bytes that are reserved but
+      do not contain data must return zero when read.
+    </t>
+
+    <t>
+      Future writes to this region MUST NOT fail with NFSERR_NOSPC.
+    </t>
+
+    <t>
+      The ALLOCATE operation will result in the space_used attribute and
+      space_freed attributes being increased by the number of bytes 
reserved
+      unless they were previously reserved or written and not shared.
+    </t>
+  </section>
+</section>
diff --git a/nfsv42_middle_op_write_plus.xml 
b/nfsv42_middle_op_write_plus.xml
index ea3f21f..85d223e 100644
--- a/nfsv42_middle_op_write_plus.xml
+++ b/nfsv42_middle_op_write_plus.xml
@@ -106,14 +106,9 @@
         deallocated.  The space_freed attribute
         may or may not decrease, depending on the support and whether the
         blocks backing the specified range were shared or not.  The size
-        attribute will remain unchanged.
-      </t>
-
-      <t>
-        The WRITE_PLUS operation MUST NOT change the space reservation
-        guarantee of the file.  While the server can deallocate the 
blocks
-        specified by di_offset and di_length, future writes to this 
region
-        MUST NOT fail with NFSERR_NOSPC.
+        attribute will remain unchanged.  If space was reserved for some
+        or all of the bytes that were converted to a hole that space
+        reservation will be dropped by WRITE_PLUS.
       </t>
     </section>
 
diff --git a/nfsv42_middle_space.xml b/nfsv42_middle_space.xml
index 44f0e40..3e2487e 100644
--- a/nfsv42_middle_space.xml
+++ b/nfsv42_middle_space.xml
@@ -4,21 +4,28 @@
 <section anchor="sec:space" title="Space Reservation">
   <section anchor="ss:space:intro" title="Introduction">
     <t>
-      Applications such as hypervisors want to be able to
-      reserve space for a file, report the amount of
-      actual disk space a file occupies, and free-up the backing space of 
a
-      file when it is not required.  In virtualized environments, virtual
-      disk files are often stored on NFS mounted volumes.  Since virtual
-      disk files represent the hard disks of virtual machines, 
hypervisors
-      often have to guarantee certain properties for the file.
+      Applications want to be able to reserve space for a file, report 
the
+      amount of actual disk space a file occupies, and free-up the 
backing
+      space of a file when it is not required.
     </t>
 
     <t>
-      One such example is space reservation.  When a hypervisor creates a
-      virtual disk file, it often tries to preallocate the space for the
-      file so that there are no future allocation related errors during 
the
-      operation of the virtual machine.  Such errors prevent a virtual
-      machine from continuing execution and result in downtime.
+      One example is the posix_fallocate(2) which allows applications to
+      ask for space reservations from the operating system, usually to
+      provide a better file layout and reduce overhead for random or
+      slow growing file appending workloads.
+    </t>
+
+    <t>
+      Another example is space reservation for virtual disks in a 
hypervisor.
+      In virtualized environments, virtual disk files are often stored on 
NFS
+      mounted volumes. 
+
+      When a hypervisor creates a virtual disk file, it often tries to
+      preallocate the space for the file so that there are no future
+      allocation related errors during the operation of the virtual 
machine.
+      Such errors prevent a virtual machine from continuing execution and
+      result in downtime.
     </t>
 
     <t>
@@ -32,13 +39,14 @@
     </t>
 
     <t>
-      We define a "reservation" as being the combination of the 
-      space_reserved attribute (see <xref target='ss:fattr:sr' />) and
-      the size attribute (see Section 5.8.1.5 of <xref target="RFC5661" 
/>).
-      If space_reserved attribute is
-      set on a file, it is guaranteed that writes that do not grow the 
file
-      past the size will not fail with NFS4ERR_NOSPC. Once the size is
-      changed, then the reservation is changed to that new size.
+      The new ALLOCATE operation (see <xref target='op:allocate' />)
+      allows a client to request a guarantee that space will be 
available.
+      The ALLOCATE operation gurantees that any future writes to the
+      region it was successfully called for will not fail with 
NFS4ERR_NOSPC.
+
+      A reservations established by ALLOCATE will be given up on a 
successful
+      cal to the WRITE_PLUS (see <xref target='op:write_plus' />) that
+      deallocates blocks.
     </t>
 
     <t>
@@ -86,11 +94,6 @@
 
     <t>
       <list style="hanging">
-        <t hangText="space_reserved">
-          This attribute specifies that writes to the reserved
-          area of the file will not fail with NFS4ERR_NOSPACE.
-        </t>
-
         <t hangText="space_freed">
           This attribute specifies the space freed when a file is
           deleted, taking block sharing into consideration.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 29 Nov 2013 23:14:13 +0000

> 14.1.3.  DESCRIPTION
> 
>    Whenever a client wishes to reserve space for a region in a file it
>    calls the ALLOCATE operation with the current filehandle set to the
>    filehandle of the file in question, and the start offset and length
>    in bytes of the region set in aa_offset and aa_length respectively.
> 
>    The server will ensure that backing blocks are reserved to the region
>    specified by aa_offset and aa_length.  If the region lies partially
>    or fully outside the current file size the file size will be set to
>    aa_offset + aa_length implicitly.  Any bytes that are reserved but do
>    not contain data must return zero when read.
> 
>    Future writes to this region MUST NOT fail with NFSERR_NOSPC.
> 
>    The ALLOCATE operation will result in the space_used attribute and
>    space_freed attributes being increased by the number of bytes
>    reserved unless they were previously reserved or written and not
>    shared.
> 

Some minor nits here and there, but they can wait.

1) What happens on creat() or open() when we truncate?

2) This becomes very posix
(see http://pubs.opengroup.org/onlinepubs/009695299/functions/posix_fallocate.html),
but we lose the ability to detect that space reservation is on for
the given file.

One argument is that stat with st_blocks allows you to determine if
the blocks are preallocated. But given that posix does not deal with
dedupped files, this can be problematic.

I.e., do we have a need to tell whether or not space reservations are set
on a file?

---

From: "Haynes, Tom" <[email-masked]>
Date: Sat, 30 Nov 2013 14:42:18 +0000

BTW - on the whole, I'm supportive of this new function.

While I believe the proposed attribute could provide the same functionality
(i.e., the attribute could mandate the same requirements about pre-allocation.),
in the end, I think the attribute isn't clear that it is really an operator.

>   not contain data must return zero when read.

What does "reserved" mean here?

Does it mean the server has to pre-allocate the blocks?

If so, are there any implications about whether ALLOCATE is synchronous or
not?

---

From: Christoph Hellwig <[email-masked]>
Date: Sat, 30 Nov 2013 17:38:17 +0100

> 1) What happens on creat() or open() when we truncate?

or a setattr that truncates for that matter, which are all the same
operation underneath.

I think the reservation for the area now lying beyond the file size
should be dropped, as the NFS spec so far seems to be very strict
about not allowing any operations outside the file size.

Note that on Linux we have a fallocate extension that allows reservations
outside the file size, but at least I don't really like the implication
of it very much becaue it now creates phantom blocks that can't easily
be found or operated on using the normal APIs.

> I.e., do we have a need to tell whether or not space reservations are set

Note that with this API reservations are always sets on regions inside
a file, not a whole file.  Because of that a query API would have to
work on regions as well.  In the context of NFSv4.2 SEEK would be the
obvious candidated to report a bool flag if a region is reserved or
not.  On a non-dedup/cow filesystem every allocated flag would have
it set, making it equivalent to the current di_allocated flag.

Given that the fact that a block that is allocated but might return
ENOSPC anyway when written to is is mostly useless to a client we
could in fact replace di_allocated with this new flag.  I'm not sure
if we'd even need to change the name or just the description.

---

From: Christoph Hellwig <[email-masked]>
Date: Sat, 30 Nov 2013 17:40:40 +0100

> Does it mean the server has to pre-allocate the blocks?

It is intentionally worded lose to either allow for a server just setting
a number of blocks aside, which I suspected was the intention of the
authors of the original space_reserved attribute, as well as an actual
allocation to unwritten extents like the Linux implementations do.

> If so, are there any implications about whether ALLOCATE is synchronous or

It is intended to be synchronous like other NFS metadata operations,
and explicitly does not include support for CB_OFFLOAD or a stable_how4
field.