Subject: [nfsv4] [NFSv4.2] Rough draft for Application Data Blocks

From: Thomas Haynes <[email-masked]>
Date: Tue, 3 May 2011 22:23:47 -0500

The following rough draft recasts the patterns from the Enterprise Apps I-D
to Application Data Blocks (ADB). It is rough as the primary intent is to answer
the concern about how corruption can be detected with a guard pattern that
does not span the entire block of data.

---

From: Dean Hildebrand <[email-masked]>
Date: Wed, 04 May 2011 11:50:53 -0700

I'm still strongly against including the pattern information in every 
READ_PLUS request, and that the XDR should 'union' it out, but I also 
realized that I don't understand why the pattern is part of the result 
(not the args) of READ_PLUS.

If a client uses INITIALIZE to create the pattern, shouldn't it then be 
also including the pattern as an argument to READ_PLUS?  The pattern 
argument is to say, "I'm reading data, but I'm also expecting a specific 
pattern.  If you see this pattern, simply return the hole as a result 
(instead of returning the pattern)".

On another note, redundantly sending the same pattern on every 
READ_PLUS, arg or result, seems wasteful.  Anything that is sent over 
and over again should be set as an well-defined attr or xattr on the file.

Dean

On 5/3/2011 8:23 PM, Thomas Haynes wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 4 May 2011 20:24:55 -0500

On May 4, 2011, at 1:50 PM, Dean Hildebrand wrote:

> I'm still strongly against including the pattern information in every READ_PLUS request, and that the XDR should 'union' it out, but I also realized that I don't understand why the pattern is part of the result (not the args) of READ_PLUS.

The point is that the client does not know whether it is going to receive either
data or a ADB.

Even if it just wrote out a set of blocks with an INITIALIZE, it cannot assume
it is getting a ADB back. It might get back data.

You have the same issue with a classic HOLE_PUNCH. You might
punch a hole and find out it only initialized the disk blocks, it did
not release them. If the client attempted to call a READ_HOLE
type of function, it would get an error.

Also, while the application which created the file and
imposed the ADB structure on it is aware of the expected format, we
cannot assume that any other client or application is aware. For example,
dd, cp, tar, cpio, rsync, scp, etc are all applications an user might
employee to backup a file. If done over NFSv3 -> NFSv4.1, these
apps will use READ, which has no knowledge of the ADB. In NFSv4.2,
it can still use the deprecated READ operation and get the same
results.

But if it uses READ_PLUS, it can leverage the sparseness that
an ADB is going to provide to reduce network traffic.

> 
> If a client uses INITIALIZE to create the pattern, shouldn't it then be also including the pattern as an argument to READ_PLUS?  The pattern argument is to say, "I'm reading data, but I'm also expecting a specific pattern.  If you see this pattern, simply return the hole as a result (instead of returning the pattern)".

Nothing dictates that the application utilizes the same sized blocks in
the file. A hypothetical application might want 4k sized blocks in the
first 100M of the file and 64k sized blocks in the next 1G of the file.
The 4k blocks are inode file blocks and the others are data blocks.

Likewise if the guard pattern is being used as a state variable, then
the client cannot assume which guard pattern it is going to get back.

> 
> On another note, redundantly sending the same pattern on every READ_PLUS, arg or result, seems wasteful.  Anything that is sent over and over again should be set as an well-defined attr or xattr on the file.

The ADB is only filled out when the server encounters a hole.

Say we create a file and initialize it with 100 4k ADBs with a block number
and guard pattern of 0xfeedface:

   INITIALIZE {0, 4k, 100, 0, 0, 8, 0xfeedface}

Further, assume the application writes a single ADB at 16k, changing the
guard pattern to 0xcafedead, we would then have in memory:

0 -> (16k - 1)       : 4k, 4, 0, 0, 8, 0xfeedface
16k -> (20k - 1)   : 00 00 00 05 ca fe de ad XX XX ... XX XX
20k -> 400k         : 4k, 95, 0, 6, 0xfeedface

And when the client did a READ_PLUS of 64k at the start of the file,
it would get back a result of an ADB, some data, and a final ADB:

ADB {0, 4, 0, 0, 8, 0xfeedface}
data 4k
ADB {20k, 4k, 59, 0, 6, 0xfeedface}

So we would require ~5k to transfer 64k of information.

The client would then be responsible for taking that result and creating
memory for the application.

This is all just sparse file semantics with a hole being defined not as
a sequential run of bytes, but rather as a sequential run of ADBs.

---

From: Thomas Haynes <[email-masked]>
Date: Fri, 6 May 2011 12:08:13 -0500

> On another note, redundantly sending the same pattern on every READ_PLUS, arg or result, seems wasteful.  Anything that is sent over and over again should be set as an well-defined attr or xattr on…

I think I understand what you are driving at here - that an ADB
is a hole which can be both stored as an unallocated disk block
and transferred as a hole. If the application does this:

And then does a

  WRITE {16k, <4k of data>}

to give us this:

Then it has to do the following in order to read the file:

READ_PLUS_FOO {0, {4k, 0, 8, 0xfeedface}}

which would mean look for a ADB that is 4k long, has the block number
at byte 0, and has the guard pattern '0xfeedface' starting at byte 8.

So if the server detected holes, it would return ADBs of the passed in
parameter. The argument then is why do we need to send the ADB at all,
the application after all knows how to map the hole to an ADB.

There are two primary reasons why we cannot simply use holes to represent
ADBs in this fashion:

1) Not every application which accesses the file knows the ADB format. [1]

tar, cp, dd, etc

2) We are not restricting a file to only have a single ADB. [2]

They could have different sizes, they could have different states
represented in the guard pattern (even as 'holes'), their block
numbers might get modified during compaction, etc.

While we could do as you suggest above and store the ADBs present in the file
as an attr/xattr, we would also have to store where within the file those
ADBs were present. In essence this would be a space map, but it would
be much more detailed.  Going to pNFS would further complicate the
accounting (which is just as true for zero filled holes).

It is simpler to return the ADB when it is encountered. I.e., the ADB is
not sent on every READ_PLUS call or result. 

Notes:

[1] And all it would take would be one NFS client that did not convert the hole
to an ADB to cause corruption at the Application level. (see for example
http://download.oracle.com/docs/cd/B28359_01/backup.111/b28270/rcmvalid.htm#CHDBFAIF
for an example of an application that defines a block of all zeros
to be corrupted)

[2] Consider that within an Oracle data block, your standard block size is 2Kb
and 'five nonstandard block sizes can be defined.' (see
http://download.oracle.com/docs/cd/B28359_01/server.111/b28318/logical.htm
for more details)