Subject: Re: [nfsv4] [pnfs] block draft clarification of layouthint

From: Benny Halevy <[email-masked]>
Date: Wed, 25 Mar 2009 09:37:39 +0200

Your use of the attribute is fundamentally different than
the file- and object- layout types, that follow 3.3.19. layouthint4:
   The layouthint4 data type is used by the client to pass in a hint
   about the type of layout it would like created for a particular file.

Rather than abusing it for this purpose, is it too late to add a suitable
attribute defined with the right scope and contents so that future
back-ends may reuse it correctly.  A Possible example is a block-based
back-end that needs the per-filesystem (or per-server) hint for client
fencing, as you suggested, plus it supports per-file layout hint, that
affects where it creates the file (think tiered storage) and how it might
be striped.

Benny

On Mar. 25, 2009, 4:34 +0200, [email-masked] wrote:
> For per-filesystem or per-server for the block layout, I think it would be per-filesystem.
> 
> The layouthint in block layout not only sends maxio_time, but also checks whether client fencing will be supported in server side.
> 
> Since data servers might differ in client fencing support, different file system might have different results for layouthint protocol.
> 
> Thanks,
> Haiying
> 
> -----Original Message-----
> From: Black, David 
> Sent: 2009年3月25日 4:59
> To: Black, David; 'Dean Hildebrand'
> Cc: [email-masked]'; [email-masked]'; Tang, Haiying; [email-masked]'; fridella, stephen
> Subject: RE: [pnfs] block draft clarification of layouthint
> Importance: High
> 
> I talked to the NFS working group chairs last night.  The upshot is
> that the scope of the layout_hint attribute could be made layout-
> specific; the text changes to do this aren't massive, but will wind
> up touching the main pNFS draft and the object layout draft in
> addition to the block layout draft.
> 
> I will raise this issue at the nfsv4 WG meeting tomorrow.  Specific
> text changes will need to be developed.  I think the changes that
> are wanted are:
> - Whether layout_hint is per-file vs. per-filesystem is pNFS layout
> 	type-specific.
> - It's per-file for the file layout
> - It's either per-filesystem or per-server for the block layout
> 	(ok, which one?)
> - It's apparently per-file for the object layout (functions as a
> 	file creation hint similar to the file layout).
> 
> Thanks,
> --David
>  
> 
>> -----Original Message-----
>> From: Black, David 
>> Sent: Friday, March 13, 2009 7:20 PM
>> To: Dean Hildebrand
>> Cc: [email-masked]; [email-masked]; Tang, 
>> Haiying; [email-masked]; fridella, stephen; Black, David
>> Subject: RE: [pnfs] block draft clarification of layouthint
>>
>>> Do you have an example of how different clients would want to set 
>>> different values?  Maybe the key point I'm missing what the possible
>>> values can be for the block layout.
>> Not offhand, unless I invent something "clever".  In most practical
>> cases there's a single value that applies to all the clients.
>>
>> Thanks,
>> --David
>>  
>>
>>> -----Original Message-----
>>> From: Dean Hildebrand [mailto:[email-masked]] 
>>> Sent: Friday, March 13, 2009 4:34 PM
>>> To: Black, David
>>> Cc: [email-masked]; [email-masked]; Tang, 
>>> Haiying; [email-masked]; fridella, stephen
>>> Subject: Re: [pnfs] block draft clarification of layouthint
>>>
>>>
>>>
>>> [email-masked] wrote:
>>>> Sorry for the delay on this - I got swamped last week by other
>>>> things.
>>>>
>>>> IMHO, we need some clarifying text, and this will need to go
>>>> to the IETF nfsv4 list soon, as I'm concerned that we may wind
>>>> up being forced to set this attribute on every file for the
>>>> block layout.
>>>>
>>>> I see two things that need to be specified.
>>>> - This is a per-file attribute that needs to be maintained
>>>> 	per-client because different clients may use different
>>>> 	values of this attribute with the same file.
>>>>   
>>> Do you have an example of how different clients would want to set 
>>> different values?  Maybe the
>>> key point I'm missing what the possible values can be for the block 
>>> layout.  This sounds like the first per-client
>>> layout related attribute that the server would have to 
>> track, or am I 
>>> forgetting something.
>>> Dean
>>>
>>>> - We can't change the attribute from a per-file attribute,
>>>> 	but we *can* specify how it defaults when it's not set.
>>>> 	The "fudge" I have in mind is that the server uses
>>>> 	the most recent value set for this attribute by this
>>>> 	client on any file as the default for files on which
>>>> 	it is not set by the client.
>>>> Does that seem reasonable?
>>>>
>>>> Thanks,
>>>> --David
>>>> ----------------------------------------------------
>>>> David L. Black, Distinguished Engineer
>>>> EMC Corporation, 176 South St., Hopkinton, MA  01748
>>>> +1 (508) 293-7953             FAX: +1 (508) 293-7786
>>>> [email-masked]        Mobile: +1 (978) 394-7754
>>>> ----------------------------------------------------
>>>>
>>>>   
>>>>> -----Original Message-----
>>>>> From: [email-masked] [mailto:[email-masked]] On 
>>>>> Behalf Of Fred Isaman
>>>>> Sent: Monday, March 02, 2009 9:38 AM
>>>>> To: Jason Glasgow
>>>>> Cc: Black, David; [email-masked]; Tang, Haiying; 
>>> fridella, stephen
>>>>> Subject: Re: block draft clarification of layouthint
>>>>>
>>>>> On Sun, Mar 1, 2009 at 9:45 PM, Jason Glasgow 
>>>>> <[email-masked]> wrote:
>>>>>     
>>>>>> We certainly want to ensure that clients only send the 
>>>>>>       
>>>>> attribute once, and
>>>>>     
>>>>>> that the server appropriately track the value per client, 
>>>>>>       
>>>>> not per file.
>>>>>     
>>>>>> That was the original intention.  (The best historical 
>>>>>>       
>>>>> record I can dig up
>>>>>     
>>>>>> is 
>>>>>>       
>> http://www.ietf.org/mail-archive/web/nfsv4/current/msg04786.html).
>>>>>     
>>>>>> In short, I think we could add some additional text stating 
>>>>>>       
>>>>> that despite the
>>>>>     
>>>>>> language of the NFSv41 spec, the block client and block 
>>>>>>       
>>>>> server MUST treat
>>>>>     
>>>>>> the layout hint as a per client attribute and not a per 
>> file, per
>>>>>> file-system or per-server attribute.
>>>>>>
>>>>>> Is there agreement with this approach?  If so, I will send 
>>>>>>       
>>>>> around a proposed
>>>>>     
>>>>>> diff.
>>>>>>
>>>>>>       
>>>>> Be careful, remember that, with the linux client, the code has to
>>>>> reconcile the block requirements
>>>>> with the generic code for all layout drivers.  The generic 
>>> client code
>>>>> has to assume it is a per-file
>>>>> attribute, preferably one sent with the opencreate 
>> attributes.  Any
>>>>> requirements the block draft
>>>>> imposes have to be sent through a framework that makes those 
>>>>> assumptions.
>>>>>
>>>>> Right now, my draft implementation does the following:
>>>>>
>>>>> In each OPEN, asks the layoutdriver(LD) for two things:
>>>>> 1. a flag, whether to encode in OPEN's createattrs, or in 
>>> a SETATTR in
>>>>> the same compound as OPEN
>>>>> 2. the (possibly empty) opaque encoding of the layouthint
>>>>>
>>>>> This defaults to the standard behavior, but gives the 
>>> block driver the
>>>>> option of sending the hint
>>>>> on the first open to a server, while ignoring other OPENs
>>>>>
>>>>> I haven't thought about the server side, but similarly, whatever
>>>>> requirement you impose needs to
>>>>> be sent through an interface designed for the 4.1 
>> expectations, not
>>>>> necessarily the block draft's.
>>>>>
>>>>> Fred
>>>>>
>>>>>     
>>>>>> -Jason
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 27, 2009 at 7:02 PM, <[email-masked]> wrote:
>>>>>>       
>>>>>>> I think Fred's interpretation is the best reading of the 
>>>>>>>         
>>>>> current text
>>>>>     
>>>>>>> - since it's a per-file attribute, it would need to be 
>>> set on every
>>>>>>> open file, and per-client indicates that different 
>>> clients may set
>>>>>>> different values (the latter is important, the former is 
>>> overkill).
>>>>>>> In an ideal world, the client would set this once 
>>>>>>>         
>>>>> attribute once and
>>>>>     
>>>>>>> it would be associated with the client ID - needless to 
>>>>>>>         
>>>>> say the world's
>>>>>     
>>>>>>> not ideal.
>>>>>>>
>>>>>>> If we want to ensure that something other than the "on 
>>> every file"
>>>>>>> implementation that Fred describes will work, I think 
>>>>>>>         
>>>>> we're going to
>>>>>     
>>>>>>> need to edit the block draft before or during Authors 48 hours,
>>>>>>> otherwise a pNFS block client will have to do roughly what Fred
>>>>>>> describes in order to deal with a server that really implements
>>>>>>> layout hint as a per-file attribute.
>>>>>>>
>>>>>>> This'll need to go to the main IETF nfsv4 list, but some 
>>> discussion
>>>>>>> here about what solution we want to see would be productive.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> --David
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ________________________________
>>>>>>> From: [email-masked] 
>>>>>>>         
>>>>> [mailto:[email-masked]] On
>>>>>     
>>>>>>> Behalf Of Jason Glasgow
>>>>>>> Sent: Sunday, February 22, 2009 11:05 PM
>>>>>>> To: Fred Isaman
>>>>>>> Cc: pNFS Mailing List; Tang, Haiying; fridella, stephen; 
>>>>>>>         
>>>>> Black, David
>>>>>     
>>>>>>> Subject: Re: block draft clarification of layouthint
>>>>>>>
>>>>>>> Fred,
>>>>>>>
>>>>>>> IIRC, the intention was that this would be a constant 
>>>>>>>         
>>>>> value per client.  I
>>>>>     
>>>>>>> would interpret "first" on a per client basis, and the 
>>>>>>>         
>>>>> meaning would be that
>>>>>     
>>>>>>> the layouthint must be sent before the first LAYOUTGET 
>>>>>>>         
>>>>> that the client
>>>>>     
>>>>>>> issues to a server.
>>>>>>>
>>>>>>> It would be valid to send the layouthint for each fh the 
>>>>>>>         
>>>>> client intends to
>>>>>     
>>>>>>> use for IO, but it might make sense to only send it once 
>>>>>>>         
>>>>> per session --
>>>>>     
>>>>>>> preceding the first LAYOUTGET issued on a new session.
>>>>>>>
>>>>>>> -Jason
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Feb 22, 2009 at 6:40 PM, Fred Isaman 
>>>>>>>         
>>>>> <[email-masked]>
>>>>>     
>>>>>>> wrote:
>>>>>>>         
>>>>>>>> I noticed some spec issues regarding layouthint while 
>>> looking over
>>>>>>>> some patches from Haiying,
>>>>>>>> and was wondering if anyone can help clarify.
>>>>>>>>
>>>>>>>> The 4.1 draft defines layouthint as a per-file attribute 
>>>>>>>>           
>>>>> that "may be
>>>>>     
>>>>>>>> set on newly created files".
>>>>>>>>
>>>>>>>> The block draft in section 2.3.8 describes its use of 
>>>>>>>>           
>>>>> layouthint to
>>>>>     
>>>>>>>> send a new blh_maximum_io_time attribute to the server.
>>>>>>>> My confusion results from the block drafts classification 
>>>>>>>>           
>>>>> of this new
>>>>>     
>>>>>>>> attribute as a "per client attribute", whereas
>>>>>>>> the 4.1 draft only has three classes, per-file, 
>>>>>>>>           
>>>>> per-filesystem, and
>>>>>     
>>>>>>>> per-server.  In particular, since it is
>>>>>>>> sent embedded in layouthint, there exist apparently 
>> conflicting
>>>>>>>> classifications of the new attribute,
>>>>>>>> as "per client" and "per file".
>>>>>>>>
>>>>>>>> So, the block draft requires that the client MUST send
>>>>>>>> SETATR(layouthint) before issuing the first LAYOUTGET.
>>>>>>>> Is this "first" on a per file basis? Or is it on a per 
>>>>>>>>           
>>>>> client basis,
>>>>>     
>>>>>>>> whatever that means?
>>>>>>>> My inclination is to try to satisfy both, by making it a 
>>>>>>>>           
>>>>> constant for
>>>>>     
>>>>>>>> the client, and sending to each fh
>>>>>>>> the client intends to use for IO.
>>>>>>>>
>>>>>>>> Is this a reasonable interpretation?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Fred
>>>>>>>>           
>>>>>>       
>>>>>     
>>>> _______________________________________________
>>>> pNFS mailing list
>>>> [email-masked]
>>>> http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs
>>>>   
>>>
> _______________________________________________
> pNFS mailing list
> [email-masked]
> http://linux-nfs.org/cgi-bin/mailman/listinfo/pnfs

---

From: [email-masked]
Date: Wed, 25 Mar 2009 11:27:28 -0400

I have slides for this morning's NFSv4 WG meeting that should be a vehicle for
raising this question.

Thanks,
--David

> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Wednesday, March 25, 2009 3:38 AM
> To: Tang, Haiying; Black, David
> Cc: [email-masked]; fridella, stephen; 
> [email-masked]; [email-masked]; NFSv4
> > The layouthint in block layout not only sends maxio_time, 
> but also checks whether client fencing will be supported in 
> server side.
> > 
> > Since data servers might differ in client fencing support, 
> different file system might have different results for 
> layouthint protocol.
> > 
> > Thanks,
> > Haiying
> > 
> > Cc: [email-masked]'; [email-masked]'; 
> Tang, Haiying; [email-masked]'; fridella, stephen