Subject: [nfsv4] READ_PLUS replies

From: Thomas Haynes <[email-masked]>
Date: Thu, 19 May 2011 15:05:37 -0500

In the meeting, we talked about sparse files and the server being able to inform the
client that a hole was larger than the data requested.

There was some confusion generated with the claim that HOLE_NOINFO was
returned to the client to indicate that this was the case.

HOLE_NOINFO is instead an error condition which either needs to belong to
NFS4ERR_* or dropped. My vote is to drop it - the case it describes does not
apply.

However, putting a semantic requirement that a READ_PLUS reply returns
the ending address of the hole makes sense.

A server gets a READ_PLUS request for 16k. If that is part of an unallocated
disk block, then it is a hole. The reply should be a hole with a starting offset
of the requested offset and a length to the first byte of real data. So if there
is a 1G hole starting at 2G into the file, and a READ_PLUS arrives for
offset 2G+32k for 16k, then the reply would be a hole starting at 2G+32k
and of length 1G-32k.

If instead of zero based holes we have ADBs, then the rule is that we always
return entire ADBs and not partial ones. So a READ_PLUS reply might start
at an offset before the request. This would probably only occur with some
application other than the owning app doing I/O.

But for the length, the same reasoning applies and we should return
all of the INITIALIZED ADBs, i.e., until the ADB state is changed to
one that does require backing storage.

For pNFS, we have a requirement that a DS only return information about
the data it owns. That does not mean that it cannot return more information
than requested.

Consider the following disk data block maps (which do not exist anywhere,
but allow us to chat) of a file of width 3:

D - data
H - hole
NI - NFS4ERR_PNFS_IO_HOLE

Dense packing:

     DS0   DS1   DS2
0  +-----+-----+-----+
   |  D  |  H  |  H  |
C  +-----+-----+-----+
   |  H  |  H  |  H  |
F  +-----+-----+-----+
   |  H  |  H  |  H  |
I  +-----+-----+-----+
   |  D  |  H  |  D  |
L  +-----+-----+-----+
   |  D  |  D  |  D  |
O  +-----+-----+-----+

Sparse packing:

     DS0   DS1   DS2
0  +-----+-----+-----+
   |  D  | NI  | NI  |
A  +-----+-----+-----+
   |  NI | H   | NI  |
B  +-----+-----+-----+
   |  NI | NI  | H   |
C  +-----+-----+-----+
   |  H  | NI  | NI  |
D  +-----+-----+-----+
   |  NI | H   | NI  |
E  +-----+-----+-----+
   |  NI | NI  | H   |
F  +-----+-----+-----+
   |  H  | NI  | NI  |
G  +-----+-----+-----+
   |  NI | H   | NI  |
H  +-----+-----+-----+
   |  NI | NI  | H   |
I  +-----+-----+-----+
   |  D  | NI  | NI  |
J  +-----+-----+-----+
   |  NI | H   | NI  |
K  +-----+-----+-----+
   |  NI | NI  | D   |
L  +-----+-----+-----+
   |  D  | NI  | NI  |
M  +-----+-----+-----+
   |  NI | D   | NI  |
N  +-----+-----+-----+
   |  NI | NI  | D   |
O  +-----+-----+-----+

And the memory addresses are 0 - Q, and each are one stripe unit size: U

The client decides to read the file and sends each DSi a READ_PLUS 
request of size U.

With the semantics of READ_PLUS, each server would reply:

Dense packing:
DS0: data of size U
DS1: hole of size 4U
DS2: hole of size 3U

Sparse packing (assume client starts at appropriate offset):
DS0: data of size U
DS1: hole of size 12U or hole of size 4U or hole of size  1U
DS2: hole of size 9U or hole of size 3U or hole of size 1U

We have different replies for the Sparse packing because it has three
possible ways to handle the hole.

1) Report a hole until the first data block. The client would have to use
its knowledge of the layout to determine which blocks corresponded
to NFS4ERR_PNFS_IO_HOLE and which ones where either ADBs
or holes.

2) Report the number of holes until the first data block. The client would
have to use its knowledge of the layout to determine which blocks
the server was sending information for.

3) Report until a NFS4ERR_PNFS_IO_HOLE is encountered.

The first approach lends itself more to what we would see back
from the dense packing. The second is more accurate about the
size of the hole. And the last means one RPC per stripe unit,
i.e., both 1) and 2) minimize network traffic.

In either packing method, the client is going to have to do a lot
of work to fill in blocks. (In the ADB case, it will also have to
manipulate the block number.)

But that savings could be huge compared to the transfer cost.

---

From: Dean <[email-masked]>
Date: Fri, 20 May 2011 14:04:35 -0700

HOLE_NOINFO is NOT an error condition, but a returned flag.

As I said in the meeting, this is a very useful optimization feature to 
reduce the number of RPCs sent to the server.

I believe the confusion lies from an incorrect copy of READPLUS from our 
draft into the NFSv4.2 draft.  Please see our original draft again 
(http://tools.ietf.org/html/draft-hildebrand-nfsv4-read-sparse-02) where 
READPLUS is defined as follows (maybe you mistook READ_OK as NFS4_OK?):

5.1. ARGUMENTS

    struct READPLUS4args {
            /* CURRENT_FH: file */
            stateid4        stateid;
            offset4         offset;
            count4          count;
    };

5.2. RESULTS

      union nfs_readplusreshole switch  (holeres4 resop) {
          CASE HOLE_NOINFO:
              void;
          CASE HOLE_INFO:
              offset4         hole_offset;
              length4         hole_length;
       };
       union nfs_readplusresok4 switch  (readplusrestype4 resop) {
          CASE READ_OK:
               opaque          data<>;
          CASE READ_HOLE:
               nfs_readplusreshole   reshole4;
       };

       union READPLUS4res switch (nfsstat4 status) {
        case NFS4_OK:
                bool            eof;
                nfs_readresok4  resok4;
        default:

Dean

On 5/19/11 1:05 PM, Thomas Haynes wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 20 May 2011 16:40:08 -0500

On May 20, 2011, at 4:04 PM, Dean wrote:

> As I said in the meeting, this is a very useful optimization feature to reduce the number of RPCs sent to the server.
> 
> I believe the confusion lies from an incorrect copy of READPLUS from our draft into the NFSv4.2 draft.  Please see our original draft again (http://tools.ietf.org/html/draft-hildebrand-nfsv4-read-sparse-02) where READPLUS is defined as follows (maybe you mistook READ_OK as NFS4_OK?):

No, I made no mistake:

From your draft there are 3 mentions of HOLE_NOINFO, here are the remaining two:

   If the server knows that the requested byte range is into a hole of
   the file, but has no further information regarding the hole, it
   returns a nfs_readplusreshole structure with holeres4 set to
   HOLE_NOINFO.
and

5.4.1. Additional pNFS Implementation Information

   With pNFS, the semantics of using READPLUS remains the same.  Any
   data server MAY return a READ_HOLE result for a READPLUS request that
   it receives.

   When a data server chooses to return a READ_HOLE result, it has a
   certain level of flexibility in how it fills out the
   nfs_readplusreshole structure.

   1. For a data server that cannot determine any hole information, the
      data server SHOULD return HOLE_NOINFO.
   2. For a data server that can only obtain hole information for the
      parts of the file stored on that data server, the data server
      SHOULD return HOLE_INFO and the byte range of the hole stored on
      that data server.

   3. For a data server that can obtain hole information for the entire
      file without severe performance impact, it MAY return HOLE_INFO
      and the byte range of the entire file hole.
As described, this is an error code and has nothing to do with an optimization
to reduce the number of RPCs.

My contention is that this should never happen.

Your text that does describe reducing the number of RPCs is as follows:

   If the client specifies an offset and count value that is entirely
   contained within a hole of the file, the status NFS4_OK is returned
   with nfs_readplusresok4 set to READ_HOLE, and if information is
   available regarding the hole, a nfs_readplusreshole structure
   containing the offset and range of the entire hole.
Where I have made a mistake was in describing the READ_PLUS to
return the requested offset plus the length. I.e., the original intent is
to return the start of the hole plus the length. I realized this right after
I sent the email yesterday.

The difference is small, but it allows read behind clients to benefit from
a hole as well as read ahead ones.

---

From: Dean <[email-masked]>
Date: Fri, 20 May 2011 15:04:05 -0700

On 5/20/11 2:40 PM, Thomas Haynes wrote:
>> As I said in the meeting, this is a very useful optimization feature 
>> to reduce the number of RPCs sent to the server.
>>
>> I believe the confusion lies from an incorrect copy of READPLUS from 
>> our draft into the NFSv4.2 draft.  Please see our original draft 
>> again 
>> (http://tools.ietf.org/html/draft-hildebrand-nfsv4-read-sparse-02) 
>> where READPLUS is defined as follows (maybe you mistook READ_OK as 
>> NFS4_OK?):
>
> No, I made no mistake:
1) You are calling HOLE_NOINFO an error code, which it is clearly 
defined as not being one in our draft.
2) You did not include HOLE_NOINFO in the definition of READPLUS in the 
latest nfsv4.2 draft, which I think is inappropriate without advance 
consultation with the original draft authors.  It should not be left to 
draft authors to dig through looking for what was cut and what was not.

The server may not know anything regarding the hole other than the 
specific read request was into a hole.  We need a way to communicate 
server knowledge of the hole to the client (once it is determined that 
the read request was into a hole.  Options are:
1) server always returns offset and length of hole given its knowledge
2) If the server has no further knowledge of the hole beyond the current 
request, server returns all zeroes for the offset and length.
3) Feedback from IETF meetings rejected #2 for having separate flags 
HOLE_NOINFO/HOLE_INFO.

We have #3 in our draft, and it should be in the NFSv4.2 draft until the 
community decides it is not needed.
Dean

>>> [email-masked] <mailto:[email-masked]>
>> [email-masked] <mailto:[email-masked]>

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 25 May 2011 01:51:15 -0500

On May 20, 2011, at 5:04 PM, Dean wrote:

> 1) You are calling HOLE_NOINFO an error code, which it is clearly defined as not being one in our draft.
> 2) You did not include HOLE_NOINFO in the definition of READPLUS in the latest nfsv4.2 draft, which I think is inappropriate without advance consultation with the original draft authors.  It should not be left to draft authors to dig through looking for what was cut and what was not.

The rough consensus in the WG had been to treat the zero filled holes as
a trivial variant of patterns (aka ADBs). As such, the last version of the
draft treats the Sparse section as being redundant. The section clearly states:

7.  Sparse Files

   WARNING: Most of this section needs to be reworked because of the
   work going on in the ADB section.

HOLE_NOINFO does not make any sense in the context of ADB. If
you combine that with the XDR errors in the Sparse draft, it got dropped.

I did listen to your argument that having data_content4 contain both
a ADB and a hole would be efficient for implementations that only
provided zero filled holes. At the last meeting, I clearly went out
of my way to point that out and to start discussions over edits for
sparse files. I brought up the point about HOLE_NOINFO. I did
not leave it for you to find.

> 
> The server may not know anything regarding the hole other than the specific read request was into a hole.  We need a way to communicate server knowledge of the hole to the client (once it is determined that the read request was into a hole.  

What is an example of when HOLE_NOINFO can be encountered?

What is the client supposed to do when it receives a HOLE_NOINFO?

It signifies a server state that does allow an operation to proceed, how is it not an error code?

> Options are:
> 2) If the server has no further knowledge of the hole beyond the current request, server returns all zeroes for the offset and     length.  
> 3) Feedback from IETF meetings rejected #2 for having separate flags HOLE_NOINFO/HOLE_INFO.
> 
> We have #3 in our draft, and it should be in the NFSv4.2 draft until the community decides it is not needed.
> Dean
> 
>> 
>>