Subject: RE: [nfsv4] Re: pnfs related comments indraft-ietf-nfsv4-minorversion1-02.txt

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 14 Jun 2006 12:57:18 -0400

With regard to comment.6, if you are ambivalent about the value of 
the feature I don't think you are the best one to write up a proposal.
I think you would have trouble then arguing that it is not a good idea,
if that's the conclusion you finally come to.

Given that Benny has said,
 
> I'm supporting doing that. It sounds appropriate as a symmetric 
> operation to a cb_layoutrecall for RECALL_FSID. 

I think the issue should be reassigned to him.

-----Original Message-----
From: Goodson, Garth 
Sent: Wednesday, June 14, 2006 12:49 PM
To: Benny Halevy
Cc: [email-masked]
Subject: [nfsv4] Re: pnfs related comments
indraft-ietf-nfsv4-minorversion1-02.txt

Benny Halevy wrote:
> There are quite a few comments in
draft-ietf-nfsv4-minorversion1-02.txt 
> that need our attention. I'm not sure which of these are tracked in
the 
> issue tracker...
> 
> Comment.1: Determine private range of layout types
> 
> I think this is important to resolve. I don't see and technical 
> difficulty with defining such a range and a mechanism (IANA?) to 
> register new layout types.
> 
I agree we need to resolve this.  I've created Issue 57 to track it.

> Comment.2: move striping information up to the generic layout level?
> 
> There doesn't seem to be much interest in doing so. I suggest removing

> this comment.
Agreed.  I'll add an issue to remove this text.

> Comment.3: it has been suggested that the HINT is a well defined type 
> other than pnfs_layoutdata4, similar to pnfs_layoutupdate4.
> 
> Is there any concrete proposal to discuss?
> 
I'm in the process of writing something up.  Issue 55 in tracker.

> Comment.4 and 5: (Resolving server callback races with sessions) We
need 
> to consider the clients' options here, and describe them...  
> NFS4ERR_DELAY has been discussed as a legal reply to CB_RECALL?
> 
> These should probably be address in conjunction with Tom's proposal.
> 
Yeah, let's see what they come up with first...  I'm going to add this 
comment to Issue 19.

> Comment.6: Should LAYOUTRETURN be modified to handle FSID callbacks?
> 
> I'm supporting doing that. It sounds appropriate as a symmetric 
> operation to a cb_layoutrecall for RECALL_FSID.
> 
This is fine, although I'm somewhat ambivalent.  It will help when read 
layouts are held, but with layouts that require layoutcommits the 
layoutreturn can be sent in the same compound as the commit.  Also, if 
we support layoutreturn by FSID then you won't be able to specify byte 
ranges -- it will have to be assumed that the whole layout is returned.

I can write up a proposal and post it for review.  I have created issue 
58 to track it.

-Garth

> Benny
> 
> Garth Goodson wrote:
>> Went through my notes and found the following from the last IETF (I 
>> think they are mostly nice-to-have):
>>
>> - possible name change for pnfs reclaim bit (a little confusing as it
is)
>> - clearer text on semantics of overlapping layouts handed out by
server
>> - best practices for when a client should do a LAYOUTCOMMIT (Trond)
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Wed, 14 Jun 2006 10:52:11 -0700

I was going to write up what the data structures would look like (i.e., 
the changes necessary to LAYOUTRETURNargs), without arguing for or against.

If Benny wants to own the issue -- or start the argument for it, that is 
fine.

-Garth

---

From: Benny Halevy <[email-masked]>
Date: Thu, 15 Jun 2006 10:42:01 +0300

> If Benny wants to own the issue -- or start the argument for it, that is 

I can take the issue...

* Arguments for having it are:
1) serve as the final LAYOUTRETURN in response to a CB_LAYOUTRECALL for FSID 
following these guidelines in section 14.5.2:

    ... A pNFS client must
    always return layout segments that comprise the full range specified
    by the recall.  Note, the full recalled layout range need not be
    returned as part of a single operation, but may be returned in
    segments.  This allows the client to stage the flushing of dirty
    data, layout commits, and returns.  Also, it indicates to the
    metadata server that the client is making progress.

    In order to ensure client/server convergence on the layout state, the
    final LAYOUTRETURN operation in a sequence of returns for a
    particular recall, SHOULD specify the entire range being recalled,
    even if layout segments pertaining to partial ranges were previously
    returned.  In addition, if the client holds no layout segment that
    overlaps the range being recalled, the client should return the
    NFS4ERR_NOMATCHING_LAYOUT error code.  This allows the server to
    update its view of the client's layout state.

2) save layoutreturn calls for read layouts

3) save layoutreturn calls for clean read/write layouts that require no flushes 
or layoutcommit

* Here's my proposed text:

22.47.  LAYOUTRETURN - Release Layout Information

22.47.1.  SYNOPSIS

    clientid, layout_type, iomode, layoutreturn, reclaim -> -

22.47.2.  ARGUMENT

    enum layoutreturn_type4 {
        RETURN_FILE = 1,
        RETURN_FSID = 2
    };

    struct layoutreturn_file4 {
        nfs_fh4         fh;
        offset4         offset;
        length4         length;
    };

    union layoutreturn4 switch(layoutreturn_type4 returntype) {
        case RETURN_FILE:
            layoutreturn_file4 layout;
        case RETURN_FSID:
            fsid4              fsid;
    };

    struct LAYOUTRETURN4args {
        clientid4               clientid;
        pnfs_layouttype4        layout_type;
        pnfs_layoutiomode4      iomode;
        layoutreturn4           layoutreturn;
        bool                    reclaim;
    };

22.47.3.  RESULT

    struct LAYOUTRETURN4res {
        nfsstat4        status;
    };

22.47.4.  DESCRIPTION

    Returns the layout segment represented by the clientid, layout type,
    iomode, filehandle and byte range or all layouts represented by the
    layout type, iomode and fsid.
    After this call, the
    client MUST NOT use the returned layout(s) and the associated storage 
protocol to
    access the file data.  The layout being returned may be a subdivision
    of a layout previously fetched through LAYOUTGET.  As well, it may be
    a subset or superset of a layout specified by CB_LAYOUTRECALL.
    However, if it is a subset, the recall is not complete until the full
    byte range has been returned.  It is also permissible, and no error
    should result, for a client to return a byte range covering a layout
    it does not hold.  If the length is all 1s, the layout covers the
    range from offset to EOF.  An iomode of ANY specifies that all
    layouts that match the other arguments to LAYOUTRETURN (i.e.,
    clientid, type, filehandle and byte range, or fsid) are being returned.

    The "reclaim" field set to "true" in a LAYOUTRETURN request specifies
    that the client is attempting to return a layout that was acquired
    before the reboot of the metadata server during the metadata server's
    grace period.   Returning layouts that were acquired during the
    metadata server's grace period MUST set the "reclaim" field to
    "false".  The "reclaim" field MUST be set to false also when layoutreturn
    is for RETURN_FSID. See LAYOUTCOMMIT (Section 22.46) for more details.

    Layouts may be returned when recalled or voluntarily (i.e., before
    the server has recalled them).  In either case the client must
    properly propagate state changed under the context of the layout to
    storage or to the server before returning the layout.

    If a client fails to return a layout in a timely manner, then the
    metadata server should use its control protocol with the storage
    devices to fence the client from accessing the data referenced by the
    layout.  See Section 14.5 for more details.

    If the layout identified in the arguments does not exist, the error
    NFS4ERR_BADLAYOUT is returned.  If a layout exists, but the iomode
    does not match, NFS4ERR_BADIOMODE is returned.

    If the LAYOUTRETURN request sets the "reclaim" field to "true" after
    the metadata server's grace period, NFS4ERR_NO_GRACE is returned.

    If the LAYOUTRETURN request sets the "reclaim" field to "true" and
    returntype is set to RETURN_FSID, NFS4ERR_INVAL is returned.

    On success, the current filehandle retains its value.
[do we need to keep this line if current fh is not used at all?]

* Also, the following sentence needs to be deleted from section 14.5.1:
  "However, there is no single operation to return all layouts associated with 
an FSID;"

Benny

> 
> -Garth
> 
> Noveck, Dave wrote:
> 
>> With regard to comment.6, if you are ambivalent about the value of the 
>> feature I don't think you are the best one to write up a proposal.
>> From: Goodson, Garth Sent: Wednesday, June 14, 2006 12:49 PM
>> To: Benny Halevy
>> Cc: [email-masked]
>> This is fine, although I'm somewhat ambivalent.  It will help when 
>> read layouts are held, but with layouts that require layoutcommits the 
>> I can write up a proposal and post it for review.  I have created 
>> issue 58 to track it.
>>
>> -Garth
>>
>>> Benny
>>>
>>> Garth Goodson wrote:
>>>

---

From: Omer Asad <[email-masked]>
Date: Thu, 15 Jun 2006 17:23:01 -0700

Hi Benny,

One small question.

>    In addition, if the client holds no layout segment that
Is there is a need to call out the case of a mis-behaving client.
If the server sends call back for a layout, then it must be thinking
that this client has a layout. so if...

a mis-behaving client, can just go ahead and keep returing
NFS4ERR_NOMATCHING_LAYOUT.

Can this just be treated a protocol violation, or should there
be some thing in the protocol to handle such an issue.

Thanks,
-Omer

> 3) save layoutreturn calls for clean read/write layouts that require no 
> flushes or layoutcommit
> 
>  "However, there is no single operation to return all layouts associated 
> with an FSID;"
> 
> Benny
> 
> 
>>
>> -Garth
>>
>> Noveck, Dave wrote:
>>
>>>> Comment.3: it has been suggested that the HINT is a well defined 
>>>> type other than pnfs_layoutdata4, similar to pnfs_layoutupdate4.
>>>>
>>> Yeah, let's see what they come up with first...  I'm going to add 
>>> this comment to Issue 19.
>>>
>>> read layouts are held, but with layouts that require layoutcommits 
>>> the layoutreturn can be sent in the same compound as the commit.  
>>> Also, if we support layoutreturn by FSID then you won't be able to 
>>> specify byte ranges -- it will have to be assumed that the whole 
>>> layout is returned.
>>>

---

From: Brent Welch <[email-masked]>
Date: Thu, 15 Jun 2006 19:06:02 -0700

I think the intent is that the server will discard its notion
that the client held a layout, so it won't keep making recalls.
It is up to the server's back-end storage to fence that client
from storage, if possible.

>>>Omer Asad said:

--
Brent Welch
Software Architect, Panasas Inc
Accelerating Time to Results(tm) with Clustered Storage

www.panasas.com
[email-masked]

---

From: Benny Halevy <[email-masked]>
Date: Fri, 16 Jun 2006 10:51:56 +0300

> that the client held a layout, so it won't keep making recalls.

Agreed.

> from storage, if possible.

Omer has a point that we may want to call out this case when the client client 
and server go out of sync. It could be a misbehaving client but it could also be 
a misbehaving (or confused) server.

I believe that this case should never happen under normal operation assuming 
that the layoutget/cb_layoutrecall race is solved.  Server recovery could be a 
bit more graceful than what Brent suggested.  After fencing the client the 
server can recall all layouts from this client (we don't have a recall all verb 
- maybe we should have one - but it could send a layoutrecall for all fsids) and 
by that reset the state on both sides of the wire, then it can unfence the 
client and resume normal operation.

Another interesting case is client recovery. If the server does not implicitly 
releases all state associated with the old incarnation of the client (as 
recommended in 14.7.2) it may send a cb_layoutrecall to the client and the 
client won't find any layout to return. I think that the client should return 
NFS4ERR_BADLAYOUT in this case rather than NFS4ERR_NOMATCHING_LAYOUT.

Benny