Subject: [nfsv4] [NFSv4.2] Meeting Notes for 05/05/2011

From: Thomas Haynes <[email-masked]>
Date: Thu, 5 May 2011 17:14:11 -0500

Attendees:

Tom Haynes (NetApp)
Fred Isaman (NetApp)
Sorin Faibish (EMC)
Peter Staubach (EMC)
Chuck Lever (Oracle)
Dean Hildebrand (IBM)
Andy Adamson (NetApp)
Trond Myklebust (NetApp)

+ IETF Note Well Agreement

   http://www.ietf.org/NOTEWELL.html

+ Administrivia

The new phone number will be used for all future meetings

+ Unstable file creation

Trond gave an overview and consensus in the meeting was we
were all interested.

We discussed recovery issues in a network partition and a server reboot.
We have existing edge case scenarios with locking, this falls into the
same camp.

AI: Trond is considering a prototype.

+ LAYOUTCOMMIT deprecation of loca_offset and loca_length

The main discussion item was whether there was an existing implementation
that utilized loca_offset and loca_length? If not, could this part of the
effort be handled in the errata.

We did not talk about the remaining part which would be done in NFSv4.2.

+ Application Data Blocks (was patterns)

Tom provided an overview of the new section sent to the WG alias.

Dean wants to have a a meeting with Application Developers to see
how they would use this for detecting corruption.

Tom stated there is plenty of OS research into detecting
memory corruption. For example, Solaris uses guard patterns
to detect kernel memory issues.

Trond chipped in that the Linux kernel also doing poisoning.

Chuck asked if we could pull in file system block corruption
references.

Tom pointed out that we have that type of experience and the
new section lays that groundwork - check sums and as Chuck
pointed out, block numbers.

Peter asked if we were solving a problem that application
developers were interested in?

Tom responded that we know of applications which do use
data blocks and which do corruption detection. What isn't
present is the efficient transfer and storage of data blocks.

Peter asked if it would be sufficient to return data and the
presence of holes. Let the client do the rest.

[Note, in going over the notes here, I see I was answering
a different question than what he was asking.
In the last meeting, we had a discussion about why couldn't
the server just return holes for the ADBs?

The reason is two-fold:

1) There can be more than one mapping of an ADB to a
sparse region. There might be several sizes of blocks
in the file, there might be several guard patterns, etc.

2) The file could be read over READ_PLUS by some
client and application other than the one which has
imposed the structure. Consider that the file is the
backing for a virtual disk and the ADBs are virtual
disk blocks. The user could turn off the VM and then
use cp, dd, tar, cpio, etc to make a backup. ]

The focus changed from corruption detection to ADB
holes and sparse files. Peter pointed out that there
was a cost to sending and storing the ADB. We have
the same approximate cost to send a zero hole as
an ADB hole, but it costs more to store an ADB hole.

Chuck pointed out that we need to consider this
along the lines of dedup. The server stores an
ADB in metadata and returns that to the NFS client.
The NFS client then expands that to present to
the application. The various parameters in the
ADB data structure dictate how the client will do so.

There was discussion about what APIs will expose
this to the application. While there would need
to be a new interface to allow an application
to INITIALIZE a file, there is no need for a new
one to allow the NFS client to return data to
the application.

Dean pointed out that we need to remove this
part from the proposed Enterprise Apps I-D and
focus just on it. Tom pointed out that was what
the document sent out a couple of nights ago did.
It focused on just the ADBs (patterns in the EA I-D)
and provided an example of how a hypothetical
application could detect corruption.

Peter wants to know an application developer
who might want to use this. He wants to understand
the difference from what is available today and
what this proposal will provide value add to the
developer.

AI: Tom to clarify the value add in the new
section, i.e., the efficient transfer and storage of
application data blocks.

Dean wants a completely fleshed out proposal
that he can forward to other people for review. He
wants to see it rewritten to specifically address
corruption. He doesn't want to see sparse reads
forced into this framework, he believes it is too complex.
He wants ADBs out of NFSv4.2 as he believes that
they are not mature enough.

Tom disagrees.

The proposal is for a generic framework to
allow:

1) corruption detection - which applications already do

2) efficient transfer and storage of data blocks - which
applications cannot do because NFS does not have
any protocol support yet.

+ NFS4ERR_NOTSUPP and enums

We did not get to this item.

---

From: Dean Hildebrand <[email-masked]>
Date: Tue, 10 May 2011 10:45:43 -0700

> they are not mature enough.

Let me clarify my concerns:
1) Lack of related work.  Why are ADBs the best method for NFS to detect 
data corruption?  What are the other possibilities (checksums, etc) and 
why are they inferior?  From the calls, I think there is a general lack 
of understanding of why this proposal is superior to the breadth of 
other data corruption detection techniques.

2) Transparency. This proposal puts all the onus on the application 
developer to detect the corruption.  Is it be possible to provide a 
end-to-end data corruption detection mechanism that is more transparent 
to the application developer?

3) Examples. So far all discussion around ADBs has been on how they work 
at the NFS level.  The motivation though seems to be to provide a data 
corruption technique for application developers.  To this point, what is 
clearly missing is a simple example of how your average C programmer 
would take advantage of ADBs.

4) An document has never existed (just emails) that answered all of 
these issues.  The original enterprise apps ID had many items in it that 
were not related to this proposal.  If such a document existed, it could 
be sent to a wider database and file system community to garner valuable 
feedback regarding data corruption techniques in distributed file systems.

5) Time. There are other much more mature proposals that were agreed to 
be in NFSv4.2.  With so many unanswered questions, and some clear 
uncertainty over the number and range of application developers that 
would take advantage of ADBs, is this really ready for the train?

Dean
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Thomas Haynes <[email-masked]>
Date: Tue, 10 May 2011 16:49:51 -0500

On May 10, 2011, at 12:45 PM, Dean Hildebrand wrote:

> 1) Lack of related work.  Why are ADBs the best method for NFS to detect data corruption?  What are the other possibilities     (checksums, etc) and why are they inferior?  From the calls, I think there is a general lack of understanding of why this proposal is superior to the breadth of other data corruption detection techniques.
> 

The section in the NFSv4.2 draft addresses this issue - the approach taken is
a generic bare bones approach that provides the framework necessary to
implement this functionality across a wide range of applications.

The example in the section also illustrates that while checksums are
one means of detecting corruption, each application might be able to
exploit rules on the state of the ADB.

One of the key points to understanding this approach is that the
guard pattern does not allow detection of corruption based on
overwrites, but as a state token. The guard pattern informs the
application on how to detect corruption in the block. And not only
can different applications have different corruption detection
algorithms, but the same application might have different approaches
in the same file.

Also, this focus on only corruption does not do service to the full
proposal, which was summarized in the notes as:

The proposal is for a generic framework to allow:

Another way to think of it is:

The ADB proposal allows applications which manage data
corruption to leverage modern storage efficiency.

> 2) Transparency. This proposal puts all the onus on the application developer to detect the corruption.  Is it be possible to provide a end-to-end data corruption detection mechanism that is more transparent to the application developer?
> 

The WG has considered checksums for NFSv4 in the past, see for example
http://www.connectathon.org/talks06/aggarwal.pdf . Some issues with that
are settling on an algorithm and the cost of computing it.

What is missing from that approach is the ability for the application to
know the state of the data and possible corruption based on that.

Consider the example from the draft, a block of data versus an indirect block.
In the first, a checksum is sufficient to find issues. In the second, whilst
a checksum might find no issues, simple checks to see if the same block
number is dereferenced multiple times or if a block number is greater
than the max available could detection application level corruption.

> 3) Examples. So far all discussion around ADBs has been on how they work at the NFS level.  The motivation though seems to be to provide a data corruption technique for application developers.  To this point, what is clearly missing is a simple example of how your average C programmer would take advantage of ADBs.

This is present in the NFSv4.2 section.

Note that this was pointed out as a deficiency in the 4/21 meeting and provided on 5/3
in time to be absorbed by the 5/5 meeting.

> 
> 4) An document has never existed (just emails) that answered all of these issues.  The original enterprise apps ID had many items in it that were not related to this proposal.  If such a document existed, it could be sent to a wider database and file system community to garner valuable feedback regarding data corruption techniques in distributed file systems.

Stating the the Enterprise Apps I-D contains unrelated items is not sufficient to
stating that these issues are unanswered. Crafting that document down to
the issues we are willing to work on is an editorial process. And that editorial
process is an ongoing effort.

The NFSv4.2 I-D had a section that reflected that editorial process prior to this meeting.
After the meeting, edits were made to help clarify the section.

If you believe a wider audience should be considering the proposal, please
point them to the document. It is out in the open. Please make them aware
that it is a work in progress.

Also, we have a choice whether to spend our energy on selling this document
or making it release ready. We had the same issue with the Labeled NFS
work and chose to focus on making it release ready. We still do not know
whether LNFS will come into NFSv4.2, but if we spend our time on making
it pretty, it will not be as functional as it can be.

> 
> 5) Time. There are other much more mature proposals that were agreed to be in NFSv4.2.  With so many unanswered questions, and some clear uncertainty over the number and range of application developers that would take advantage of ADBs, is this really ready for the train?

The original Enterprise Apps I-D presented a comprehensive approach to
sparse files (reading and writing), hole punching, and space reservations.
Each of which were considered important on their own.

Each of these items still have open issues that need to be addressed, either
in the context of ADBs or on their own. I.e., does a hole punch go to the
MDS or the DS? If it goes to the MDS, does it force the layout to be
recalled?

Our criteria for the time driven release model is how much energy is someone
willing to apply - not how much energy is it going to take. The way that will
be enforced is our commitment to drop something from the release because it
impacts the schedule.

Finally, what is NFSv4.2 supposed to be? Is it a vehicle to introduce major
infrastructural changes like NFSv4.0 and NFSv4.1?

No.

I believe the focus on NFSv4.2 is to bridge the gap between applications,
the protocol, and storage. We know storage systems can support sparse
files, hole punches, space reservations, and server to server copy. And
we know applications will perform much better if these features can
be exploited during data transfer. We also know applications can
detect data corruption at the application and not OS level. We need to
exploit both ends of the spectrum.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 11 May 2011 10:57:37 -0400

Hi-

> than the max available could detection application level corruption.

To put it another way, detecting corruption in a shared storage system can involve detecting a variety of failure modes.  Mis-ordered updates to a data block -- application timing or serialization problems.  Or incomplete updates to a data block -- torn pages.  Or updates to the wrong block.

Or lost writes.  If a write is lost, a data block checksum on disk, based on a previous write, is potentially correct, but the contents of the block are semantically incorrect because they do not contain the expected data.  Allowing an application to add information to the "hole" means it can detect lost writes using redundant metadata (see Tom's example above).

Another problem with checksums is where the sum is computed.  Does the application compute it?  Does the storage? If the storage computes it, how are transport failures or mis-ordered updates detected, for example?  Is the checksum strong enough?  If the storage handles it, then the application has no choice about the checksum strength or where the checksum is computed.  If the application wants a stronger checksum, then potentially both storage and applications checksum their data, which is wasteful.

For instance, some levels of RAID compute parity on data blocks.  But that doesn't matter at all if a buggy file system implementation writes blocks in the wrong location.  The file system on disk is clearly corrupt, but the underlying RAID subsystem doesn't complain; the individual blocks on disk all match their individual block checksums and data parity blocks.

Thus a more "transparent" or generalized corruption detection system cannot benefit from application-specific knowledge to detect data corruption.  The purpose of something like ADB is to enable the entire storage stack (application to media and back) to detect corruption at any stage.  This is critical as storage stacks become increasingly more complex, more virtualized, and more redundant.

Keep in mind that the term "application" can mean any software that wants to store data durably and reliably: a database, a file system, an emulated block device, and so on.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 18 May 2011 13:30:40 -0500

> than the max available could detection application level corruption.

Spencer pointed me to an interesting white paper from EMC: EMC Double Checksum for Oracle

Google found me a copy here:
http://68-191-184-102.static.fdul.wi.charter.com/doc/vendors/emc/emc-docs/symcli6.0/whpapers/3000208h.pdf

The paper is from 2004 and is concerned with SCSI, but it does highlight
an application's data view being imposed on storage. The storage layer
recomputes the block checksum on writes and compares it against the
one computed by the application.

The approach does not introduce ADB, but it does show that indirectly that is done:

File system-based Oracle files are not currently supported by EMC Double
Checksum. All Oracle tablespace files must be placed on raw volumes (including
LVM raw volumes). If host-based striping is used on the raw volumes, then the
stripe size must be an exact multiple of the Oracle block size.

I.e., the paper does not talk about how the OS gets the Oracle computed checksum,
but it has to grab it at a known offset in the block.

The downside to such an approach is that the checksum is fixed into an application
specific location and the algorithm for providing the checksum is application specific.

The ADB approach solves this with its generic framework. It describes the block
such that the server does not have to provide a raw disk and allows the location
to be specified by the application. Whilst we could add a location for a checksum
and start an IANA registry up for the ids for checksum algorithms, such an
approach does not allow for application specific tests to occur. I.e., ones based
on the state of the block. The guard pattern allows for that.

But this framework we are presenting does allow for end-to-end checksums
to be built on top of it. We could define a new ADB format in a future version
and it could provide the location, length, and algorithm needed to compute
the end-to-end checksum.

The ADB approach can be thought of as a virtual disk in a file. We provide
a framework which allows any application to describe that virtual disk,
to allow quick initialization of it, and to detect corruption based on the
state of the block.

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 18 May 2011 14:36:24 -0500

Another effort being driven in part by Oracle is presented here:

Eliminating Silent Data Corruption with Oracle Linux
http://oss.oracle.com/~mkp/docs/data-integrity-webcast.pdf

This effectively adds 8 bytes (called a PI) to each 512 byte data block. 

Those 8 bytes are:

16 bit CRC of the 512 bytes
16 bit application tag
32 bit reference tag

While neither the application or reference tag are
defined in this document, they are defined in http://oss.oracle.com/~mkp/docs/dix-draft.pdf

1) The guard tag protects the data portion of the sector.
2) The application tag is simply opaque storage.
3) And finally, the reference tag is being used to protect against out-of-order and misdirected write scenarios.

I.e., the application tag is our guard pattern and the
reference tag is our block number.

The 8 bytes do not need to be stored in the same disk block as
the data payload. They do need to come across the wire together.

This approach could be defined in terms of an ADB. 

We could also adopt the T10 Protection Information Model outright.
We would want to still be able to initialize each 520 bytes
to be consistent. I.e., 512 bytes are zeroed, 16 bit is zeroed,
16 bits are set to initialized state, and reference tag is set.
The ADB would shrink, but the same basic functionality would
work.

---

From: Dean Hildebrand <[email-masked]>
Date: Wed, 18 May 2011 12:58:04 -0700

> to allow quick initialization of it, and to detect corruption based on the
ADB can only detect corruption in 'empty' blocks correct?  How does it 
detect corruption is blocks that have data in them?

Dean

---

From: Thomas Haynes <[email-masked]>
Date: Wed, 18 May 2011 15:08:59 -0500

On May 18, 2011, at 2:58 PM, Dean Hildebrand wrote:

> ADB can only detect corruption in 'empty' blocks correct?  How does it detect corruption is blocks that have data in them?
> 
> Dean
>> 
>> 
>> 
>> 

The application needs to initialize blocks in a file to a known state. Until
actual data is written to a block, we represent it via an ADB. And that
is reflected as the data_content4 having the value of
NFS4_CONTENT_APP_BLOCK.

Once the application changes the state of the ADB and writes to
it, the client will send a WRITE operation back to the server.

Whenever the client reads the block back in, it will do so with
READ_PLUS again, but this time the data_content4 will have
the value of NFS4_CONTENT_DATA.

The application will then see that the guard pattern is in
a different state than INITIALIZED and will be able to
detect whatever corruption is called for by that state.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 19 May 2011 03:56:22 +0000

Thanks for finding the reference, Tom.

Note that we have had other proposals for end-to-end integrity features.

It may be worth a short review of that and other approaches (as Tom has done)
because as the proposal receives broader review, these questions and
comparisons will arise.

Spencer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Thomas Haynes
> Sent: Wednesday, May 18, 2011 12:36 PM
> To: [email-masked]
> Subject: Re: [nfsv4] End-to-end corruption checks
> 
> While neither the application or reference tag are defined in this
> document, they are defined in http://oss.oracle.com/~mkp/docs/dix-
> draft.pdf
> 
> I.e., the application tag is our guard pattern and the reference tag is
> our block number.
> 
> The 8 bytes do not need to be stored in the same disk block as the data
> payload. They do need to come across the wire together.
> 
> We would want to still be able to initialize each 520 bytes to be
> consistent. I.e., 512 bytes are zeroed, 16 bit is zeroed,
> The ADB would shrink, but the same basic functionality would work.

---

From: Nico Williams <[email-masked]>
Date: Thu, 19 May 2011 12:02:43 -0500

The ZFS (and BTRFS, and... Tahoe? and...?) experience is instructive,
as it has end-to-end integrity protection, but only where the ends are
the filesystem implementation itself, not the application.

In the case of NFS the filesystem resides on the server, so the
checksums in the filesystem approach doesn't help -- in the NFS case
we want the ends to be the clients.  But this complicates matters.

I see two possible _generic_ approaches:

1) Clients can read file _data_ block checksums (really, strong
hashes), but only for aligned data blocks of whatever size the
server-side filesystem prefers.  For meta-data it's a bit more
complicated, and for directory contents this approach is utterly
painful.

2) The server-side filesystem does local end-to-end integrity
protection but also computes checksums (hashes) for the clients to
see, with the server validating the on-disk hashes [again] _after_
computing the hashes for the clients.  This approach makes alignment
and block size issues go away.  For meta-data and READDIR just use
integrity protection of the compound op results (and skip this if
RPCSEC_GSS is being used with data integrity protection) -- this
approach means there's a tiny window for cosmic ray bit flipping in
memory on the server-side, but only for meta-data, not data.  When
writing data the client would also send data checksums.

My preference is for (2), as it's the most generic solution.  Both can
result in APIs to expose checksums to apps, but neither has to.

Nico
--

---

From: <[email-masked]>
Date: Thu, 19 May 2011 15:06:42 -0400

> This effectively adds 8 bytes (called a PI) to each 512 byte data block.

Those 8 bytes are standardized in the SCSI standards as "Protection Information" - see Section 4.19 in the Revision 26 working draft of the SBC-3 standard at www.t10.org).  Hence if this format of check data is used by the application, it can be checked below the server filesystem when SCSI Protection Information support is present in the block storage infrastructure.  Some adjustment is needed as the check passes through layers (e.g., to remap the reference tag from offset in file to LBA offset in logical unit [storage volume] - see Section 1.4 of the dix draft linked in Tom's message below).

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Thomas Haynes
> Sent: Wednesday, May 18, 2011 3:36 PM
> To: [email-masked]