Subject: [nfsv4] Proposal for integrating application data holes into the traditional file model

From: Christoph Hellwig <[email-masked]>
Date: Thu, 7 Nov 2013 21:26:33 +0100

1) Rationale

The NFSv4.2 specification introduces a new data content abstraction
that extends the previous flat file concept used in previous NFS
specifications.

The two new content types are holds and application data holes (ADHs).
While the whole is trivially backwards compatible the application
data hole concept creates various interoperability problems.  This
proposal tries to come up with a way to provide the functionality
of ADHs without introducing a new data format and thus avoiding these
problems, as well as making the functionality provided by ADHs available
to a broader set of users.

2) ADH Issues

NFSv4.2 clients that support the READ_PLUS operation must be able to
handle a ADH returned from the server at any time.  Because efficient
sparse file support requires the READ_PLUS operation and the READ
operation has been marked obsolete іt is expected that most NFSv4.2
clients will use READ_PLUS and thus will have to provide a way to
expand ADHs to flat file data as expected by most clients.

NFS servers will still need to support the READ command and thus also
have to provide a mechanism to expand ADHs to flat file data.

An algorithm for the expansion of ADHs to flat file data is not
provided as part of the NFSv4.2 specification.

A NFSv4.2 client wishing to find the next bit content cannot use
the new SEEK operation as that would require specifying if it is looking
for traditional flat data or an ADH before doing the seek call.  It could
mitigate that by doing a seek for both types of data and looking for
the lower offset, although this would be slightly awkward.

ADHs can not be exposed by pNFS layout types other than NFSv4.2 files
as neither the NFSv4.1 data server protocol, nor the T10 SBC or OSD2
protocols support a similar concept, nor is a mapping for those
protocols specified.

3) Solution

It appears the ADH concept developed from the INITIALIZE operation
proposed in 

	http://tools.ietf.org/html/draft-eisler-nfsv4-enterprise-apps-01

which has none of the above issues.  The INITIALIZE operation is modeled
after the T10 SBC WRITE SAME family of operations which have traditionally
been used by enterprise application, hypervisors and file systems to provide
the functionality that ADHs attempt to provide.

By dropping the high-level ADH concept in the NFSv4.2 and providing a
functionality that mirrors the proposed INITIALIZE operation and is
similar to the SBC WRITE SAME operations the following benefits can
be archived:

 - because the mapping to flat file blocks is specified no special case
   is required on the read side.  Clients are isolated from how the
   file was initialized, support for old clients on the read or overwrite
   side is automatically provided, and pNFS integration works seamlessly.

 - servers that use SBC backing devices can translate the the NFS operation
   into the T10 operation easily.

 - hypervisors or storage targets that use NFS for their backing storage
   can easily implement the T10 WRITE SAME operations on top of NFS.

 - the INITIALIZE operation can be used to zero out large amounts of space
   even if the underlying device does not support hole punching or efficient
   preallocation, and thus in combination with the SPACEMGMT proposal
   will allow to fully replace the NFS4_CONTENT_HOLE: arm of the WRITE_PLUS
   operation

If the INITIALIZE operation gets added to the NFSv4.2 specification it
should support the offload mechanism specified for WRITE_PLUS.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 8 Nov 2013 18:41:50 +0000

> provided as part of the NFSv4.2 specification.

In the context of ADH, the server is expected to create blocks of
adh_block_size, with adh_block_num at adh_reloff_blocknum,
adh_pattern at adh_reloff_pattern. The rest of the block will be zeros.

> for traditional flat data or an ADH before doing the seek call.

An ADH is taken to be a hole in this scenario.

> the functionality that ADHs attempt to provide.

Despite being in discussion for over 2 years, no one has pointed out the
use of WRITE SAME before.

I don’t see a good description online about how it will handle the block
number and pattern. Do you have a reference?

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 8 Nov 2013 20:11:48 +0100

> An ADH is taken to be a hole in this scenario.

The way I read 14.11.2. SEEK gets a "data_content4 sa_what" as part
of the argument, which can be either NFS4_CONTENT_DATA,
NFS4_CONTENT_APP_DATA_HOLE or NFS4_CONTENT_HOLE, and returns
a app_data_hole4 if it called for NFS4_CONTENT_APP_DATA_HOLE.

I do not see any language special casing ADHs.  But if they were special
cased they should not be treated as a hole because they do contain data.

Think about it - Poor Admin Joe does a backup of Big Larry's database,
which makes heavy used of ADHs.  The production server goes boom so
it now needs the backup from the nfs volume at the end of the universe.
If the copy utility using SEEK skipped over the ADHs because they were treated
like holes Big Larry would not be a happy camper.

If on the other hand ADHs disappear as special case of file content in
the interface and only live on inside the nfs server at the end of the
universe life would be fine.

> number and pattern. Do you have a reference?

Grab a copy of the SBC standard.  Unfortunately the current ones aren't
publicly available since T10 went closed doors, but
ftp://www.t10.org/t10/document.05/05-344r0.pdf is recent enough for dicussing
the basic WRITE SAME features.

The payload to the WRITE SAME command is the pattern, which gets written
into each LBA.  Unlike the ADH proposal it is not specified as a sub-region,
but leaving most of the pattern as zero will archive the same effect in
a slightly less efficient format.  The block number handling is a lot
less flexible, but look at Table 75 on Page 42 for the few two options
to put the logical block number in.

The protection information is a fairly recent feature that hasn't been
available for a long time, but Chuck might be chime in as he's been looking
into similar end-to-end protection features for NFS.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 8 Nov 2013 20:18:50 +0000

> cased they should not be treated as a hole because they do contain data.

Okay, I agree that an ADH cannot be handled via SEEK as is.

I also agree that if the only questions are DATA or HOLE, that we should
return DATA.

And I’m okay with that for systems trying to live with POSIX style glasses.

But we should be able to expose that it is an ADH to non-POSIX clients
that want to do SEEK_ADH.

So how about:

14.11.3.  DESCRIPTION

   From the given sa_offset, find the next data_content4 of type sa_what
   in the file.  For either a NFS4_CONTENT_HOLE or
   NFS4_CONTENT_APP_DATA_HOLE, this must return the
   data_content4 in its entirety.  For NFS4_CONTENT_DATA, it must not return the
   actual data.

   If sa_what is NFS4_CONTENT_APP_DATA_HOLE then when the server
   encounters an ADH, then it MUST reply with a NFS4_CONTENT_APP_DATA_HOLE.
   However if the sa_what is not NFS4_CONTENT_APP_DATA_HOLE, then
   when the server encounters an ADH, it MUST return a result consistent with
   the ADH being evaluated as if it were NFS4_CONTENT_DATA. I.e., a sa_what
   of NFS4_CONTENT_HOLE will skip over the ADH and a NFS4_CONTENT_DATA
   will return the ADH as if it were data.
   
   SEEK must follow the same rules for stateids as READ_PLUS
   (Section 14.10.3).

   If the server could not find a corresponding sa_what, then the status
   would still be NFS4_OK, but sr_eof would be TRUE.  The sr_contents
   would contain a zero-ed out content of the appropriate type.

---

From: Christoph Hellwig <[email-masked]>
Date: Fri, 8 Nov 2013 21:42:51 +0100

> Okay, I agree that an ADH cannot be handled via SEEK as is.

I think SEEK is the smallest problem, in fact I only added it to my problem
statement in the last minute before sending it.  The existance of ADHs
is a major problem for Posix-like clients, and probably just about any
client that isn't called "Oracle Database".

The dicussion I wanted to start with the problem statement is to find
someone to defend the model and actually show real-world benefits of
exposing the structured blocks vs being able to write them, but accessing
them as a normal data stream.  Given that even the original authors of
the enterprise applications proposal seemed to think the way I do I think that
is a very reasonable question.

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 8 Nov 2013 23:18:12 +0000

On Nov 8, 2013, at 12:42 PM, Christoph Hellwig <[email-masked]> wrote:

> Given that even the original authors of

Given the those authors had not seen the glory of READ_PLUS, you can
understand how they did not write their approach with that in mind.

You have two choices with structured blocks:

1) Take the otw call and write the blocks to disk.

Benefit: Save OTW call on INITIALIZE

Cons: Reading requires you to get the now instantiated disk blocks.

2) Store the structured blocks as extended attributes of the file
such that the server can regenerate the data blocks as needed.

Benefit: Save OTW call on INITIALIZE and READ_PLUS. Keep a sparse file.

Cons: Need to associate an extended attribute, “map” fragments as blocks
are instantiated by later WRITEs by the clients, handling hole punches of
different sized blocks into existing ADHs, having different sized blocks.

And the rough consensus within the WG was that the benefits of 2 outweighed
the cons. :-)

---

From: Christoph Hellwig <[email-masked]>
Date: Sat, 9 Nov 2013 09:40:59 +0100

> 1) Take the otw call and write the blocks to disk.

OTW = on the wire?

> And the rough consensus within the WG was that the benefits of 2 outweighed

This discussion missed two important points:

 - even if the data model is defined as returning the pattern in the
   regular flat file format that does not impose any internal data
   storage on the server.
 - given that the NFSv4.2 sparse file model is not a block-based one,
   but an arbitrary byte range on a server implementing the pattern
   initialization by storing metadata does not have to return a full
   block worth of mostly zeroes, even if sticking to the flat file
   mode.

And of course it totally ignores the much bigger implications changing
the file model has on clients that have to deal with the new concept
even if they have absolutely no use for, the pnfs implications and everthing
else I wrote down in the first mail.

Interesting nugget from the spec btw:

"If the client asks for a hole and
 the server does not support that arm of the discriminated union, but
 does support one or more additional arms, it can signal to the client
 that it supports the operation, but not the arm with
 NFS4ERR_UNION_NOTSUPP."

This seems to imply there every server that supports holes would also
support ADHs.  That's for sure not true for the Linux server with Annas
hole punch patches, and I also expect it to not be true for most
other servers sitting on top of traditional Unix filesystems.

---

From: "Noveck, David" <[email-masked]>
Date: Sat, 9 Nov 2013 13:41:08 -0500

> the server does not support that arm of the discriminated union, but  does support one or more additional arms, it can signal >  to the client  that it supports the operation, but not the arm with  NFS4ERR_UNION_NOTSUPP."

> This seems to imply there every server that supports holes would also support ADHs.

I don't read it that way.   Specifically, the case of a server that supports holes and not ADH's
Is just one more (good) reason to return NFS4ERR_UNION_NOTSUPP.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christoph Hellwig
Sent: Saturday, November 09, 2013 3:41 AM
To: Haynes, Tom
Cc: Christoph Hellwig; nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Proposal for integrating application data holes into the traditional file model

OTW = on the wire?

> 2) Store the structured blocks as extended attributes of the file such 
> that the server can regenerate the data blocks as needed.
> 
> Cons: Need to associate an extended attribute, “map” fragments as 
> blocks are instantiated by later WRITEs by the clients, handling hole 
> punches of different sized blocks into existing ADHs, having different sized blocks.
> 
> And the rough consensus within the WG was that the benefits of 2 
> outweighed the cons. :-)

And of course it totally ignores the much bigger implications changing the file model has on clients that have to deal with the new concept even if they have absolutely no use for, the pnfs implications and everthing else I wrote down in the first mail.

"If the client asks for a hole and
 the server does not support that arm of the discriminated union, but  does support one or more additional arms, it can signal to the client  that it supports the operation, but not the arm with  NFS4ERR_UNION_NOTSUPP."

This seems to imply there every server that supports holes would also support ADHs.  That's for sure not true for the Linux server with Annas hole punch patches, and I also expect it to not be true for most other servers sitting on top of traditional Unix filesystems.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, David" <[email-masked]>
Date: Sat, 9 Nov 2013 14:14:20 -0500

Christoph said:
> The existance of ADHs is a major problem for Posix-like clients, and probably just about any client that isn't 
> called "Oracle Database".

The place this problem becomes manifest is, for example, when you copy a database containing ADH's.   Unless,
you think that "Every time I read this file, my kernel panics" is a great database security feature :-), there has to 
be some way for a non-ADH-aware client to read a file containing ADH's from an ADH-supporting server.  Without that, 
ADH-supporting servers are likely to be few and far between, since they won't interoperate with a large
class of clients.  

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Christoph Hellwig
Sent: Friday, November 08, 2013 3:43 PM
To: Haynes, Tom
Cc: nfsv4 list ([email-masked])
Subject: Re: [nfsv4] Proposal for integrating application data holes into the traditional file model

> Okay, I agree that an ADH cannot be handled via SEEK as is.

I think SEEK is the smallest problem, in fact I only added it to my problem statement in the last minute before sending it.  The existance of ADHs is a major problem for Posix-like clients, and probably just about any client that isn't called "Oracle Database".

The dicussion I wanted to start with the problem statement is to find someone to defend the model and actually show real-world benefits of exposing the structured blocks vs being able to write them, but accessing them as a normal data stream.  Given that even the original authors of the enterprise applications proposal seemed to think the way I do I think that is a very reasonable question.