Subject: Re: [nfsv4] dense and sparse with IO_ADVISE (was Re: [NFSv4.2] Meeting Minutes for 09/22/2011

From: Dean <[email-masked]>
Date: Thu, 06 Oct 2011 09:43:09 -0700

Thanks Mike.  I agree with your text.

In addition, are we saying that for clients sending the IO_ADVISE operation, if dense layouts are used then IO_ADVISE will use dense byte ranges? (and conversely if NFL42_UFLG_IO_ADVISE_THRU_MDS is set or NFL4_UFLG_DENSE is not set, then IO_ADVISE will use 'posix/file/sparse' byte ranges?

Dean

On 9/30/11 1:56 PM, Mike Eisler wrote:
> On Thu, September 22, 2011 10:25 pm, Thomas Haynes wrote:
>
>> + IO_ADVISE
>>
>> Dean summarized Mike's new text.
>>
>> He had some questions about dense versus sparse.
> I believe the issue is this. If both of the following are true:
> - the pNFS server is allowing IO_ADVISE to a DS
> - the layout is sparse
>
> then the client can potentially specify a byte range that hits an illegal
> hole in the DS's file. We should examine two operations, READ and WRITE,
> that also take a byte range, and can be sent to DSes.
>
> If we sent a WRITE to a sparse DS file that hits an illegal hole, we would
> return an error, NFS4ERR_PNFS_IO_HOLE. If we sent a READ to a sparse DS
> file that hits an illegal hole, then if the offset were in the hole, we
> would return NFS4ERR_PNFS_IO_HOLE. If the offset were not in the hole, but
> the byte range, due to length, hit a hole, the server could simply return
> a short read instead of NFS4ERR_PNFS_IO_HOLE, but nonetheless it could
> return NFS4ERR_PNFS_IO_HOLE.
>
> So the question is under what conditions the server return
> NFS4ERR_PNFS_IO_HOLE to IO_ADVISE on sparse layouts? There are two
> extremes
> we can take here, and a spectrum of approaches in between.
>
> At one extreme, which I will call the strict extreme, the DS rejects with
> NFS4ERR_PNFS_IO_HOLE any IO_ADVISE that specifies a byte range that hits a
> hole. In general this is bad because if the client wants to for example
> indicate that sequential access is going to be used, then it just wants to
> send an offset of zero, and a length of zero, which indicates sequential
> access for the entire file. The strict extreme would require many
> IO_ADVISE requests, one per stripe unit. For small stripe units, this is
> going to be really bad.
>
> At the other extreme, which will call the laid back extreme, the DS
> never rejects any byte range with the error NFS4ERR_PNFS_IO_HOLE.
>
> I suggest the laid back extreme because it produces the fewest IO_ADVISE
> operations.
>
> So I propose the following text to be added to the I-D:
>
> "If neither of the flags NFL42_UFLG_IO_ADVISE_THRU_MDS and NFL4_UFLG_DENSE
> are set in the layout, then any IO_ADVISE request sent to the data server
> with a byte range that overlaps stripe unit that the data server does not
> serve MUST NOT result in the status NFS4ERR_PNFS_IO_HOLE. Instead, the
> response SHOULD be successful and if the server applies IO_ADVISE hints on
> any stripe units that overlap with the specified range, those hints SHOULD
> be indicated in the response.
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: <[email-masked]>
Date: Thu, 6 Oct 2011 14:06:54 -0400

First of all, I agree that the strict extreme is not good.

I'm worrying that the laid back extreme is, well, too extreme. 

The case that concern me is if the byte-range is totally outside the set of data that the DS serves.  Maybe this issue applies to both dense and sparse layouts..  

What happens in PNFS if I send  READ or WRITE to the wrong DS and the layout is dense.  I assume it is an error but I couldn't figure out from a very quick look at RFC5661 what error it would be.

So the question I have is separate from dense vs. sparse.  If you send an IO_ADVISE to a Ds and no byte in your range is covered by that ds, should there be an error, what error should it be, and should it be the same whether your range crosses an illegal hole or not?

My answers are; yes, I don't know, and yes.

Other opinions?

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 6 Oct 2011 18:21:56 -0700

Dean,

Yes, IO_ADVISE has to use the offset and byte range as dictated by the
presence or absence of NFL4_UFLG_DENSE .

E.g. So if NFL4_UFLG_DENSE is present, and a READ or WRITE to the DS for
offset zero really means offset 10000 in the logical file, then an
IO_ADVISE for offset zero means offset 10000.

E.g. So if NFL4_UFLG_DENSE is absent, then a READ or WRITE to the DS for
offset zero really means offset zero in the logical file, then an
IO_ADVISE for offset zero means offset zero in the logical file.

E.g. If NFL4_UFLG_DENSE is present, the stripe unit is 1000 bytes and the
stripe count is 10, and the dense DS file is serving offset zero. A  READ
or WRITE to the DS for offsets zero, 1000, 2000, and 3000, really mean
offsets 10000, 20000, 30000, and 40000 (implying a stripe count of 10 and
a stripe unit of 1000), then an IO_ADVISE sent to the same DS with an
offset of 500, and a count of 3000 means that the IO_ADVISE applies to
these byte ranges of the dense DS file:
- 500 to 999
- 1000 to 1999
- 2000 to 2999
- 3000 to 3499

I.e. the contiguous range 500 to 3499 as specified in IO_ADVISE.

and these byte ranges of the logical file:

- 10500 to 10999 (500 bytes)
- 20000 to 20999 (1000 bytes)
- 30000 to 30999 (1000 bytes)
- 40000 to 40499 (500 bytes)

(total            3000 bytes)

E.g. If NFL4_UFLG_DENSE is absent, the stripe unit is 250 bytes, the
stripe count is 4, and the sparse DS file is serving offset zero. Then a
READ or WRITE to the DS for offsets zero, 1000, 2000, and 3000, really
mean offsets zero, 1000, 2000, and 3000 in the logical file, keeping in
mind that on the DS file,. byte ranges 250 to 999, 1250 to 1999, 2250 to
2999, and 3250 to 3999 are not accessible. Then an IO_ADVISE sent to the
same DS with an offset of 500, and a count of 3000 means that the
IO_ADVISE applies to these byte ranges of the logical file and the sparse
DS file:

- 500 to 999 (500 bytes) - no  effect
- 1000 to 1249 (250 bytes) - effective
- 1250 to 1999 (750 bytes) - no effect
- 2000 to 2249 (250 bytes) - effective
- 2250 to 2999 (750 bytes) - no effect
- 3000 to 3249 (250 bytes) - effective
- 3250 to 3499 (250 bytes) - no effect

(subtotal      2250 bytes) - no effect
(subtotal       750 bytes) - effective

(grand total   3000 bytes) - no effect + effective

On Thu, October 6, 2011 9:43 am, Dean wrote:
> In addition, are we saying that for clients sending the IO_ADVISE
> operation, if dense layouts are used then IO_ADVISE will use dense byte
> ranges? (and conversely if NFL42_UFLG_IO_ADVISE_THRU_MDS is set or
> NFL4_UFLG_DENSE is not set, then IO_ADVISE will use 'posix/file/sparse'
> byte ranges?
>
> Dean
>

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: "Mike Eisler" <[email-masked]>
Date: Thu, 6 Oct 2011 18:43:29 -0700

inline ...

On Thu, October 6, 2011 11:06 am, [email-masked] wrote:
> The case that concern me is if the byte-range is totally outside the set
> of data that the DS serves.  Maybe this issue applies to both dense and
> sparse layouts..
>
> What happens in PNFS if I send  READ or WRITE to the wrong DS and the
> layout is dense.  I assume it is an error but I couldn't figure out from a
> very quick look at RFC5661 what error it would be.

Dense packing: There's little way to detect the situation you describe. 
If the client miscalculates the destination DS,
a WRITE will corrupt the file, and the READ will return what looks like
corrupted data to the application or its user (if either have any data
corruption checking mechanism).

One might mitigate the problem is if the server returns a unique
filehandle for each DS file. If the client miscalculates the DS network
address, but not the DS filehandle (or does the converse), then DS will
reject the operation as a stale filehandle. But that type of bug seems
unlikely to me. DENSE is dangerous. I've never liked dense packing, but
that's not new information.

Sparse packing: This situation is probably detectable.
If the client miscalculates the destination DS, then a
a WRITE will be rejected with NFS4ERR_PNFS_IO_HOLE, a READ with an
unserved offset will be rejected with NFS4ERR_PNFS_IO_HOLE, and a READ
with a served offset, but unserved byte range, will be a short READ.
Sparse is good. I've always liked sparse, but that's not new information.

> So the question I have is separate from dense vs. sparse.  If you send an
> IO_ADVISE to a Ds and no byte in your range is covered by that ds, should
> there be an error, what error should it be, and should it be the same
> whether your range crosses an illegal hole or not?
>

My answers are no, N/A, and N/A.

I say no because I don't want an error from IO_ADVISE to cause a COMPOUND
to fail. I think an NFSv4.2 server SHOULD always support IO_ADVISE, even
if
if it always returns a zeroed out, or zero length hint array. Some of the
hints work best on a per READ or WRITE basis, and so IO_ADVISE will prefix
the READ or WRITE. I want the READ or WRITE to be attempted.

Smart client/stupid server. :-)

---

From: Dean <[email-masked]>
Date: Tue, 11 Oct 2011 14:49:59 -0700

I've posted a new version of the IO Advise draft that includes Mike's 
latest text in this email thread.  Please let me know if I missed or 
misquoted something.

http://tools.ietf.org/html/draft-hildebrand-nfsv4-ioadvise-02

Dean

===
A new version of I-D, draft-hildebrand-nfsv4-ioadvise-02.txt has been 
successfully submitted by Dean Hildebrand and posted to the IETF repository.

Filename:  draft-hildebrand-nfsv4-ioadvise
Revision:  02
Title:  Support for Application IO Hints
Creation date:  2011-10-11
WG ID:  Individual Submission
Number of pages: 16

Abstract:
   This document proposes a new IO_ADVISE operation for NFSv4.2 that
   clients can use to communicate expected I/O behavior to the server.
   By communicating future I/O behavior such as whether a file will be
   accessed sequentially or randomly, and whether a file will or will
   not be accessed in the near future, servers can optimize future I/O
   requests for a file by, for example, prefetching or evicting data.
   This operation can be used to support the posix_fadvise function as
   well as other applications such as databases and video editors.

The IETF Secretariat