Subject: [nfsv4] SEEK di_allocated meaning

From: Anna Schumaker <[email-masked]>
Date: Fri, 1 Nov 2013 14:56:21 -0400

Hi again,

The SEEK operation returns a data_info4 structure as part of the result.  Does the "di_allocated" field have any meaning for this operation?  As far as I can tell, di_allocated == true when seeking for data and false when seeking for a hole.  Am I missing something?

struct data_info4 {
   offset4        di_offset;
   length4        di_length;
   bool           di_allocated;
};

Anna

---

From: "Haynes, Tom" <[email-masked]>
Date: Fri, 1 Nov 2013 19:26:28 +0000

> The SEEK operation returns a data_info4 structure as part of the result.  Does the "di_allocated" field have any meaning for this operation?  As far as I can tell, di_allocated == true when seeking…

A hole can either be in the file or it can be in what the server would send.

I.e., if the server finds a range of byte which are all zeros, it can report a hole.

This allows a server which is using a filesystem which does not support sparse files to
avoid the transfer of all zeros.

---

From: Marc Eshel <[email-masked]>
Date: Fri, 1 Nov 2013 16:51:17 -0700

I tough that di_allocated is used to indicate if the hole is part of a 
file that is allocated. Even if the serve support sparse file the range 
that it reports back can be part of the file that is zero because it the 
sparse part, but even the sparse part can be allocated or not.
Marc.

From:   "Haynes, Tom" <[email-masked]>
To:     "Schumaker, Bryan" <[email-masked]>, 
Cc:     "[email-masked]" <[email-masked]>
Date:   11/01/2013 12:26 PM
Subject:        Re: [nfsv4] SEEK di_allocated meaning
Sent by:        [email-masked]

On Nov 1, 2013, at 11:56 AM, Anna Schumaker <[email-masked]> wrote:

> Hi again,
> 
> The SEEK operation returns a data_info4 structure as part of the result. 
 Does the "di_allocated" field have any meaning for this operation?  As 
far as I can tell, di_allocated == true when seeking for data and false 
when seeking for a hole.  Am I missing something?
> Anna

A hole can either be in the file or it can be in what the server would 
send.

I.e., if the server finds a range of byte which are all zeros, it can 
report a hole.

This allows a server which is using a filesystem which does not support 
sparse files to
avoid the transfer of all zeros.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, David" <[email-masked]>
Date: Sat, 2 Nov 2013 08:26:08 -0400

Tom's explanation still leaves me confused about this issue.  If you say that's my
natural state :-), then you're probably right :-(

> I.e., if the server finds a range of byte which are all zeros, it can report a hole.

If the range were reported as a hole, it would't be reported as data with di_allocated false.

> This allows a server which is using a filesystem which does not support sparse files to avoid the 
> transfer of all zeros.

But to avoid the transfer of zeros, it has to report a hole, and so the server for that filesystem 
would support the reporting of sparse files but not the creation of holes. 

My speculation is that di_allocated being false might arise in a complementary situation.  Suppose 
you have a filesystem that does support sparse files, as a result of hole-punching or simply sparse 
writes.  Such a filesystem might be confronted with the writing of zero blocks and simply treat them 
as holes internally but not report them as holes, because that state is transient (i.e. it can be 
changed by writes upto one block away and the client doesn't know the fs block size) .   Such an 
internal transformation could be considered and implemented as a form of compression or of 
dedup.  It might make sense to report such regions as data with di_allocated false.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Friday, November 01, 2013 3:26 PM
To: Schumaker, Bryan
Cc: [email-masked]
Subject: Re: [nfsv4] SEEK di_allocated meaning

This allows a server which is using a filesystem which does not support sparse files to avoid the transfer of all zeros.

---

From: Marc Eshel <[email-masked]>
Date: Sat, 2 Nov 2013 10:59:48 -0700

I don't understand the spec or the explanations that I read so far so let 
me try again, and correct me if I am wrong. 
The hole can have 3 states:
1. the hole is allocated but full of zeros so di_allocated is true (the 
di_ prefix is confusing it sounds like de-allocated)
2. the hole is part of a sparse file that have no blocks assigned for this 
range so di_allocated is false 
3. the hole is part of a sparse file and space was reserved for this file 
so di_allocated is true 
Marc.

From:   "Noveck, David" <[email-masked]>
To:     "Haynes, Tom" <[email-masked]>, "Schumaker, Bryan" 
<[email-masked]>, 
Cc:     "[email-masked]" <[email-masked]>
Date:   11/02/2013 05:26 AM
Subject:        Re: [nfsv4] SEEK di_allocated meaning
Sent by:        [email-masked]

Tom's explanation still leaves me confused about this issue.  If you say 
that's my
natural state :-), then you're probably right :-(

send.

report a hole.

If the range were reported as a hole, it would't be reported as data with 
di_allocated false.

> This allows a server which is using a filesystem which does not support 
sparse files to avoid the 
> transfer of all zeros.

But to avoid the transfer of zeros, it has to report a hole, and so the 
server for that filesystem 
would support the reporting of sparse files but not the creation of holes. 

My speculation is that di_allocated being false might arise in a 
complementary situation.  Suppose 
you have a filesystem that does support sparse files, as a result of 
hole-punching or simply sparse 
writes.  Such a filesystem might be confronted with the writing of zero 
blocks and simply treat them 
as holes internally but not report them as holes, because that state is 
transient (i.e. it can be 
changed by writes upto one block away and the client doesn't know the fs 
block size) .   Such an 
internal transformation could be considered and implemented as a form of 
compression or of 
dedup.  It might make sense to report such regions as data with 
di_allocated false.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of 
Haynes, Tom
Sent: Friday, November 01, 2013 3:26 PM
To: Schumaker, Bryan
Cc: [email-masked]
Subject: Re: [nfsv4] SEEK di_allocated meaning

This allows a server which is using a filesystem which does not support 
sparse files to avoid the transfer of all zeros.

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 3 Nov 2013 21:36:12 +0000

> di_ prefix is confusing it sounds like de-allocated)

The 'i' is not silent.

The prefix is a common format we started as of NFSv4.1.

> 3. the hole is part of a sparse file and space was reserved for this file 

The third is incorrect.

di_allocated indicates solely whether or not the hole currently is allocated.

---

From: "Haynes, Tom" <[email-masked]>
Date: Sun, 3 Nov 2013 21:40:22 +0000

> If the range were reported as a hole, it would't be reported as data with di_allocated false.

Never said it would be - but for argument's sake, I will profess to being sparse in my reply.

di_allocated is always true for data.

It may or may not be true for a hole.

---

From: "Noveck, David" <[email-masked]>
Date: Mon, 4 Nov 2013 06:24:48 -0500

> Tom said:

> Never said it would be - but for argument's sake, I will profess to being sparse in my reply.

But your reply is not full of zeros, does not seem to be a hole, and I'm not clear if di_allocated
would be set :-)

> di_allocated is always true for data.

That makes sense but note that what started this was the natural assumption that if it is being
reported for data, it must mean something in that context.

>> The SEEK operation returns a data_info4 structure as part of the result.  Does the "di_allocated" field have 
>> any meaning for this operation?  As far as I can tell, di_allocated == true when seeking for data and false 

So now I note that SEEK returns data_info4's in two arms of the result, so taking Tom's answer in account, and
leaving ADH's out of it:

     sc_data_di_allocated is always (MUST be?) true.

     sc_hole.di_allocated may be true or false, but it isn't clear to me why it is useful. 

The other thing unclear about di_allocated is its use in WRITE_PLUS.   Is wpa_hole.di_allocated just ignored? 

-----Original Message-----
From: Haynes, Tom [mailto:[email-masked]] 
Sent: Sunday, November 03, 2013 4:40 PM
To: Noveck, David
Cc: Schumaker, Bryan; [email-masked]
Subject: Re: [nfsv4] SEEK di_allocated meaning

On Nov 2, 2013, at 5:26 AM, Noveck, David <[email-masked]> wrote:

> Tom's explanation still leaves me confused about this issue.  If you 
> say that's my natural state :-), then you're probably right :-(
> 

> 
>> This allows a server which is using a filesystem which does not 
>> support sparse files to avoid the transfer of all zeros.
> 
> But to avoid the transfer of zeros, it has to report a hole, and so 
> the server for that filesystem would support the reporting of sparse files but not the creation of holes.
> 
> complementary situation.  Suppose you have a filesystem that does 
> support sparse files, as a result of hole-punching or simply sparse 
> writes.  Such a filesystem might be confronted with the writing of zero blocks and simply treat them as holes internally but not report them as holes, because that state is transient (i.e. it can be
> internal transformation could be considered and implemented as a form 
> of compression or of dedup.  It might make sense to report such regions as data with di_allocated false.
>

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 4 Nov 2013 16:27:39 +0000

On Nov 4, 2013, at 3:24 AM, Noveck, David <[email-masked]<mailto:[email-masked]>> wrote:

On Nov 1, 2013, at 11:56 AM, Anna Schumaker <[email-masked]<mailto:[email-masked]>> wrote:

Saul had wanted it, so perhaps he can share the use case.

I guess it can be used as a hint.

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 4 Nov 2013 18:31:10 +0000

On Nov 4, 2013, at 8:27 AM, Haynes, Tom <[email-masked]<mailto:[email-masked]>> wrote:

14.10.3

   If the data to be returned is comprised entirely of zeros, then the
   server may elect to return that data as a hole.  The server
   differentiates this to the client by setting di_allocated to TRUE in
   this case.  Note that in such a scenario, the server is not required
   to determine the full extent of the "hole" - it does not need to
   determine where the zeros start and end.  If the server elects to
   return the hole as data, then it can set the d_allocted to FALSE in
   the rpc_data to indicate it is a hole.

14.7.3.2.  Hole punching

   Whenever a client wishes to zero the blocks backing a particular
   region in the file, it calls the WRITE_PLUS operation with the
   current filehandle set to the filehandle of the file in question, and
   the equivalent of start offset and length in bytes of the region set
   in wpa_hole.di_offset and wpa_hole.di_length respectively.  If the
   wpa_hole.di_allocated is set to TRUE, then the blocks will be zeroed
   and if it is set to FALSE, then they will be deallocated.  All
   further reads to this region MUST return zeros until overwritten.
   The filehandle specified must be that of a regular file.

In the READ_PLUS case, I think it should be called out as a SHOULD.

In the WRITE_PLUS case, I think we should stress that it is a hint which
the server is free to ignore.

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 4 Nov 2013 22:05:38 +0000

On Nov 4, 2013, at 2:09 AM, Christoph Hellwig <[email-masked]> wrote:

> On Sun, Nov 03, 2013 at 09:40:22PM +0000, Haynes, Tom wrote:
> In addition how do you expect a client to use it?  None of the syscall
> interface expose this amount of detail.

I actually expect most clients to ignore it.

We had a use case, we rolled it in.

> 
> Also how does this fit into the general model of space allocation in the
> NFS 4.2 spec?  The only way to make it fit would be to require di_allocated
> to be set if the hole is accounted in the space_used attributed,

Why would we have to require it?

I view the space allocation and this flag to be orthogonal.

If space allocation is set, then regardless of whether di_allocated is set or not,
the blocks represented by the hole are accounted for by the size of the file.

If it is not set, then di_allocated tells us whether or not the blocks
are represented in the size.

> but the
> problem with that is that the existing syscall level interfaces as well as
> the local filesystem implementations I'm aware of generally do not expose
> the level of data allocation for a hole, making this difficult to
> report.
> 

> Based on the tho rationales I would strongly recommend dropping this
> flag from the specification.

---

From: "Haynes, Tom" <[email-masked]>
Date: Tue, 5 Nov 2013 02:58:41 +0000

On Nov 4, 2013, at 5:01 PM, Christoph Hellwig <[email-masked]> wrote:

> To have a coherent model of space allocations.
> 
> So if it does not represent the same model of space allocations as the
> space_alloc flag what use does the allocate flag have?

What space_alloc flag?

That is not in the spec. (Yes, I am aware it is probably something specific
to the Linux client.)

Both di_allocated and d_allocated indicate whether or not there are any
backing blocks to the hole or data respectively.

Regardless of whether or not they are set, nothing changes with the space
reservation.

If space reservation is set on a file, then the server must guarantee that
any operation which fills the hole with data will not fail with NFSERR_NOSPC.

> 
> What model of space allocations do you expect a client to have,

*I* don't have a model of how clients handle space allocations. We had
client implementors engaged in the discussions and they didn't have
any objections.

Don't get me wrong here, more than happy to fold in implementation
experience into the spec.

> and how
> do the differing views of space_used and di_allocated fit into?

See the first paragraph of Section 14.7.3.2 of Draft 21. The last two
also apply.

I've already admitted I am not happy with that definition. At best it
should be a hint.

> 
> Given that the 4.2 draft introduce this notion of allocated space we'd
> better make sure to have a coherent model of it instead of exposing
> random bits related to space allocations that might contradict each
> other.

Given I believe we have a coherent model, let me ask this differently.

1) Is such a flag of any use other than informational when coming back
from READ_PLUS?
a) Given the current APIs available to clients?
b) Or is there something worth it to go modify the APIs?

2) For WRITE_PLUS:
a) If not a hint, does the server reject the operation if it cannot support
the request?
b) Even if a hint, is there anyway for the client to tell whether or not
the server conclusively acted on the request?

If it can't pass a use case analysis, it can go.

---

From: Saul Tamari <[email-masked]>
Date: Tue, 5 Nov 2013 12:28:06 -0500

Hi,

I think it was a colleague of mine who asked about this a few years
back so I talked to him and he came up with this use case:
"This could be useful when we want to to copy a DB file, but want to
keep the DB allocation policy so it wont be thin provisioned"

Saul

On Mon, Nov 4, 2013 at 11:27 AM, Haynes, Tom <[email-masked]> wrote:
>
> On Nov 4, 2013, at 3:24 AM, Noveck, David <[email-masked]> wrote:
>
>
> Does the "di_allocated" field have

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 6 Nov 2013 02:26:59 +0000

> keep the DB allocation policy so it wont be thin provisioned"

Is this not solved best by using space reservation and not caring
about whether or not the server is using sparse files?

Also, there can be no guarantee that the server will honor the
request to not thin provision.

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 6 Nov 2013 04:33:58 +0000

> On Nov 5, 2013, at 6:30 PM, "Haynes, Tom" <[email-masked]> wrote:
> 
> 

Further, can't you examine the space_reserved attribute on the original file to determine the DB allocation policy?

---

From: Saul Tamari <[email-masked]>
Date: Thu, 7 Nov 2013 12:23:26 -0500

COPY would work also for servers from different vendors?

On Wed, Nov 6, 2013 at 2:32 AM, Christoph Hellwig <[email-masked]> wrote:
> On Tue, Nov 05, 2013 at 12:28:06PM -0500, Saul Tamari wrote:
>> Hi,
>>
> COPY should take care of that for intra-server copies at least without
> exposing allocation details.
>

---

From: "Haynes, Tom" <[email-masked]>
Date: Thu, 7 Nov 2013 19:25:47 +0000

On Nov 7, 2013, at 10:10 AM, Christoph Hellwig <[email-masked]> wrote:

> 
> But longer term I have to agree that reporting the allocation status
> would be useful, and adding protocol support for it might be useful
> as well, thus changing my earlier opinion.  For it to be useful it
> would have to be exact reporting, through.  One possible way out of
> that dilema would be to change the allocation state from a bool to
> a tristate and allow servers to report that they don't know the
> allocation state.  This would allow to implement fully conforming
> servers ontop of the existing syscall or VFS APIs while leaving the
> option open to report the more detailed state if available.
> 

I can see the client giving the server a hint of I don’t care how you
allocate the file, but the server should always be able to reply with
whether the hole is allocated or not.

---

From: Christoph Hellwig <[email-masked]>
Date: Thu, 7 Nov 2013 20:34:39 +0100

On Thu, Nov 07, 2013 at 07:25:47PM +0000, Haynes, Tom wrote:
> whether the hole is allocated or not.---end quoted text---

I'd like to agree with you, but in practice many NFS servers sit behind
interfaces they don't control, and they might not make this amount
of lowlevel information available, especially as the standards track
lseek SEEK_HOLE/SEEK_DATA interface doesn't offer it.

---

From: "Haynes, Tom" <[email-masked]>
Date: Thu, 7 Nov 2013 19:58:19 +0000

> lseek SEEK_HOLE/SEEK_DATA interface doesn't offer it.

How does the NFS server decide between the definitive answer
and the don't know?

If the interfaces do not support querying the state, then the server
should reply don't know?

What if the interface does not support this, but the NFS server folds
the zeros into a hole?

---

From: Christoph Hellwig <[email-masked]>
Date: Thu, 7 Nov 2013 21:13:57 +0100

> If the interfaces do not support querying the state, then the server

Yes.

> What if the interface does not support this, but the NFS server folds

Assuming the server implements the other parts of sparse file support
and it has already skipped over an actual hole using SEEK_DATA it knows
that it just read data and may set di_allocated.

But that and the language in the speak says in the READ_PLUS section
makes me a bit more worried.

Let me enumerate the possible cases we have to deal with here:

 1) an actual unallocated hole in the file

 2) an unwritten extent, that is an extent allocated on disk,
    that contains invalid data, but that the file system returns
    zero for on reads.  Aka the PNFS_BLOCK_INVALID_DATA in
    pNFS block layouts.

 3) an actual data extent where the file system or server detected
    that data is zeroed.

For 1) the answer is easy: we should not set di_allocated, and if the
file system interface supports detecting sparse files it can always
communicate this.

For 2) the answer should be that di_allocated is set, although the
wording in the standard does not seem to include this case.  Detecting
this isn't possible with the existing SEEK_DATA/SEEK_HOLE system call
interface, although file systems generally have this information available.
As a historical anecdote the first implementations of SEEK_DATA/SEEK_HOLE
reported this as data in Linux, but do report it as a hole now.

For 3) the standard says it must be set.  If the NFS server does the hole
detection itself this is easily doable, if the file system does it on the
read side the same communication problem as in 2 exists.

Based on this one way out might be wording that uses SHOULD and MAY
and takes greater care of the above cases:

"If rpa_offset points into a region that does not have blocks allocated
 to it the server SHOULD return a hole that does not have di_allocated
 set.
 If rpa_offset points into a region that does have blocks allocated but
 that would return all zeros the server MAY return a hole with
 di_allocated set".

Note that this also takes care of the case that the read request may
be for a region that is partially covered by a hole and partially by
data, in which case we would have to return a short read for the hole,
something that doesn't seem to be handled in the current spec.

I would also advocate for exposing the same information in SEEK as
in the READ_PLUS hole case for consistency reasons.