Subject: [nfsv4] proposal for loga_length and loga_minlength

From: "Mike Eisler" <[email-masked]>
Date: Fri, 2 May 2008 11:21:11 -0700 (PDT)

Problems:

- loga_minlength as specified is useless
- the rules for when a server can return a layout of a range that
  is a super-set, sub-set, intersection, ... of the requested range
  (loga_offset, ... loga_offset + loga_length - 1) are unclear

- Some people have expressed a desire for an "opportunistic" LAYOUTGET
  that returns a layout if the amount of overhead on the pNFS server is
  minimal (e.g. if it does not require provisiong storage devices and/or
  space on storage devices).

Proposal:

- The layout returned MUST have an offset that is no higher than
  loga_offset

- The layout returned MUST have a length that is no smaller than
  loga_minlength.

- loga_length MUST be >= loga_minlength

- loga_minlength MAY be zero (0).

The idea is that loga_offset, loga_minlength indicate the layout the client
"desires", but loga_offset, loga_minlength indicate the layout the client
"requires".

E.g.:

   SEQUENCE, OPEN, LAYOUTGET {off=0, length=NFS4_UINT64_MAX, minlength=0 }

This tells the server to try to give the client a layout for the full
range 0 ... 2^64-1 but if that's not available, give it what is
available, and if that's
not even immediately available, give it NFS4ERR_LAYOUTTRYLATER.

Some client's might at least want something, so they do:

      SEQUENCE, OPEN, LAYOUTGET {off=0, length=NFS4_UINT64_MAX, minlength=1 }

The tells the server to try to give the client a layout for range
0 ... 2^64-1, but if that's not immediately available, give it
0 ... 0 (or a range that goes beyond offset 0).

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 02 May 2008 13:30:22 -0500

> The idea is that loga_offset, loga_minlength indicate the layout  

loga_minlength should be loga_length.

> 0 ... 0 (or a range that goes beyond offset 0).

The above is reasonable.

Spencer

---

From: Robert Gordon <[email-masked]>
Date: Fri, 02 May 2008 13:41:19 -0500

It's clear, no dot-x changes, and ASSERT(everyone likes it).

Robert.

---

From: [email-masked]
Date: Fri, 2 May 2008 14:47:30 -0400

The proposal makes sense to me. -Jason

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Spencer Shepler
> Sent: Friday, May 02, 2008 2:30 PM
> To: Mike Eisler
> Cc: [email-masked]
> Subject: Re: [nfsv4] proposal for loga_length and loga_minlength
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 3 May 2008 13:43:58 -0400

Works for me. 

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Friday, May 02, 2008 2:41 PM
To: NFSv4
Subject: Re: [nfsv4] proposal for loga_length and loga_minlength

Robert.

On May 2, 2008, at 1:30 PM, Spencer Shepler wrote:

>> "desires", but loga_offset, loga_minlength indicate the layout the 
>> client "requires".
>>
>> E.g.:
>>

>> available, and if that's not even immediately available, give it 
>> NFS4ERR_LAYOUTTRYLATER.
>>
>> The tells the server to try to give the client a layout for range 0 
>> ... 2^64-1, but if that's not immediately available, give it 0 ... 0 
>> (or a range that goes beyond offset 0).
>
> The above is reasonable.
>
> Spencer

---

From: Benny Halevy <[email-masked]>
Date: Sun, 04 May 2008 13:56:24 +0300

>> - The layout returned MUST have an offset that is no higher than

like I previously said here:
http://www.ietf.org/mail-archive/web/nfsv4/current/msg04948.html
I think that the returned range should cover loga_offset but
not necessarily start at it, since the server may want to align
the starting offset differently.

>> - The layout returned MUST have a length that is no smaller than

I'm sort of OK with that, although I'd be happier with a SHOULD
rather than MUST.

If, for alignment reasons logr_offset < loga_offset and assuming,
as you mention below that the client requires the range
log_offset to log_offset + loga_minlength (non-inclusive)
than the returned lo_offset + lo_length SHOLUD be >=
loga_offset + loga_minlength.
But for the same alignment constraints lo_length might be shorter.

Now that the logr_layout is an array, the server can describe
the rest of the region in the next entry, but:
a. loga_maxcount must be large enough... and that concerns me
especially with widely striped files, and
b. The layout for the next stripe or group of stripes may be
held by another client and I would like to do the CB_LAYOUTRECALL
to the other client concurrently with this client's I/O.

In summary, the client doesn't have to use the layout and can
return it and stop asking for more layouts if it doesn't like the
layouts the server gives it, so IMO loga_minlength should be
treated as a hint rather than a hard requirement, otherwise, I
anticipate clients will reduce it to something quite small,
e.g. 1, or one page size.

We can certainly recommend that the client not be greedy and send
the smallest loga_minlength it can live with...

>> - loga_length MUST be >= loga_minlength

That makes sense.

>> - loga_minlength MAY be zero (0).

I like that too (well, I need to be consistent with what I
suggested myself in the past ;-).

Benny

>> client

---

From: "Mike Eisler" <[email-masked]>
Date: Sun, 4 May 2008 09:45:43 -0700 (PDT)

My first attempt at this proposal as stated has a bug where
if a server returned an offset < loga_offset, e.g 10 versus 100, and
if loga_minlength were 25, then the server could return a layout of
offset 10, and length 25 (which doesn't even overlap the requested
layout). So the corrected proposal below:

Problems:

Proposal:

- If loga_minlength is     NFS4_UINT64_MAX, the layout returned MUST have a
  length that is NFS4_UINT64_MAX.

- If loga_minlength is not NFS4_UINT64_MAX, the layout returned MUST have a
  length that is no smaller than
        loga_offset - returned offset + loga_minlength.

The idea is that loga_offset, loga_length indicate the layout the client
"desires", but loga_offset, loga_minlength indicate the layout the client
"requires".

E.g.:

   SEQUENCE, PUTFH, OPEN, LAYOUTGET {off=0, length=NFS4_UINT64_MAX,
minlength=0 }

      SEQUENCE, PUTFH, OPEN, LAYOUTGET {off=0, length=NFS4_UINT64_MAX,
minlength=1 }

To acknowledge the bug in the original proposal, more examples:

      SEQUENCE, PUTFH, LAYOUTGET {off=1000, length=NFS4_UINT64_MAX,
          minlength=1 }

Client can for example, return a layout with:
    offset 500, length 501

------------

      SEQUENCE, PUTFH, LAYOUTGET {off=1000, length=NFS4_UINT64_MAX,
          minlength=NFS4_UINT64_MAX }

Client can for example, return a layout with:
    offset 50, length NFS4_UINT64_MAX

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 4 May 2008 13:43:49 -0400

Since LAYOUTGET can return an array of layouts, I think the text needs
to be written with that in mind.  The offset of the set of layouts is
the offset of the first and the length of the set is the sum of the
lengths of the individual layouts.

I believe your approach is basically right but there are a number of
issues related to NFS4_UNIT64_MAX (abbreviated u64m below) that may
cause editors-niggling-detail-frustration syndrome (not in the DSM yet).
No results of controlled tests of treatments for this condition have
been reported although there is anecdotal evidence that single-malt
scotch can be quite effective :-)

     The requirement that the returned length be u64m causes
     causes difficulties when offset is above one.  You can
     create a rule to allow offset+length to be greater than
     u64m+1, but requiring that u64m be used in this case
     (as opposed to u64m+1-offset) seems unnatural and this
     is even worse when you are adding up offsets of multiple
     layouts in an array are added up.

     Dealing with this is complicated by the fact that the 
     "natural" range with an offset of zero and a length of 
     u64m does not include the byte at offset u64m.

If loga_minlength is u64m then the layout or layouts returned must cover
the byte at offset u64m.

If loga_minlength is not u64m, the sum of loga_offset and loga_minlength
must not exceed u64+1, INVAL if it does.

Otherwise, as you have it, except that you consider sum of lengths when
multiple layouts are returned.

When a layouts has the length u64m, it is to be interpreted as the value
necessary to make the layout cover the byte at offset u64m.  For layouts
with offset zero, this is a length of u64m+1.

When layout lengths to get total lengths, the length value u64m is not
to be used directly but rather the value that it represents.

Except in the case of a length of u64m, the offset plus the length of
specific layout MUST ot exceed u64m+1. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Sunday, May 04, 2008 12:46 PM
To: [email-masked]
Subject: Re: [nfsv4] proposal for loga_length and loga_minlength

My first attempt at this proposal as stated has a bug where if a server
returned an offset < loga_offset, e.g 10 versus 100, and if
loga_minlength were 25, then the server could return a layout of offset
10, and length 25 (which doesn't even overlap the requested layout). So
the corrected proposal below:

Problems:

Proposal:

- If loga_minlength is     NFS4_UINT64_MAX, the layout returned MUST
have a
  length that is NFS4_UINT64_MAX.

- If loga_minlength is not NFS4_UINT64_MAX, the layout returned MUST
have a
  length that is no smaller than
        loga_offset - returned offset + loga_minlength.

E.g.:

The tells the server to try to give the client a layout for range 0 ...
2^64-1, but if that's not immediately available, give it 0 ... 0 (or a
range that goes beyond offset 0).

------------

---

From: Robert Gordon <[email-masked]>
Date: Sun, 04 May 2008 12:54:38 -0500

> the rest of the region in the next entry, but:

the current wording is such that there is only one element
in the array when there is a single stripe pattern.

are you saying that we should change that ??

Robert..