Subject: Re: [nfsv4] pnfs: read/write through MDS hint (Issue 55)

From: Garth Goodson <[email-masked]>
Date: Mon, 31 Jul 2006 13:28:11 -0700

I've updated my proposal for the hint to help the client determine 
whether or not it should issue I/O through the MDS.

This proposal is for a new attribute: pnfs_mdsthreshold.

This attribute acts as a hint to the client to help it determine when
it is more efficient to issue read and write requests to the metadata
server vs. the dataserver.  Two types of thresholds are described:
file size thresholds and I/O size thresholds.  If a file's size is
smaller than the file size threshold, data accesses should be issued
to the metadata server.  If an I/O is below the I/O size threshold,
the I/O should be issued to the metadata server.  Each threshold can
be specified independently for read and write requests.  For either
threshold type, a value of 0 indicates no read or write should be
issued to the metadata server, while a value of ~0 indicates all reads
or writes should be issued to the metadata server.

The attribute is available on a per filehandle basis.  If the current
filehandle refers to a non-pNFS file or directory, the metadata server
should return an attribute that is representative of the filehandle's
file system.  It is suggested that this attribute is queried as part
of the OPEN operation.  Due to dynamic system changes, the client
should not assume that the attribute will remain constant for any
specific time period, thus it should be periodically refreshed.

Structures:
-----------

struct pnfs_mdsthreshold_item {
   pnfs_layouttype4  layout_type;
   bitmap4           hintset;
   opaque            hintlist<>;
};

struct pnfs_mdsthreshold {
   pnfs_mdsthreshold_item hints<>;
};

The pnfs_mdsthreshold structure holds an array of hints that can help
the client determine when it should issue I/O directly through the
metadata server vs. the data servers.  Each hint structure is specific
to a layout type.  Since an implementation may support multiple layout
types, an array of these structure is needed.  The hint structure
consists of the layout type, a bitmap describing the set of hints
supported by the server, they may differ based on the layout type, and
a list of hints, whose structure is determined by the hintset bitmap.

Bitmap descriptions for the hintlist field:

Name                   #  Data Type Access  Description
----                   -  --------- ------  -----------
read_size_threshold    1  length4   READ    The file size below which it
                                             is recommended to read data
                                             through the MDS.

write_size_threshold   2  length4   READ    The file size below which it
                                             is recommended to write data
                                             through the MDS

read_iosize_threshold  3  length4   READ    For read I/O sizes below
                                             this threshold it is
                                             recommended to read data
                                             through the MDS

write_iosize_threshold 4  length4  READ     For write I/O sizes below
                                             this threshold it is
                                             recommended to write data
                                             through the MDS

Marc Eshel wrote:
> I would be happy with the simple struct if we added a separate attribute 
> for read and write.
> 
>  struct pnfs_mdsthreshold_item {
>     pnfs_layouttype4  layout_type;
>     length4           read_size_threshold;
>     length4           write_size_threshold;
>  };
> 
> Marc.
> 
> Garth Goodson <[email-masked]> wrote on 06/14/2006 03:35:15 PM:
> 
>> There has been some discussion about a hint that can be used to help a 
>> client decide whether or not to read/write through the MDS vs the DS. 
>> I've described two types that could be used for these hints.  The first 
>> is fairly basic, while the second is a little more complex.  The 
>> motivation for the second is to provide a way to extend this attribute 
>> in the future.
>>
>> 1. Simple structure:
>> --------------------
>>
>> struct pnfs_mdsthreshold_item {
>>    pnfs_layouttype4  layout_type;
>>    length4           size_threshold;
>> };
>>
>> struct pnfs_mdsthreshold {
>>    pnfs_mdsthreshold_item hints<>;
>> };
>>
>>
>> 2. More complex, more extensible:
>> ---------------------------------
>>
>> struct pnfs_mdsthreshold_item {
>>    pnfs_layouttype4  layout_type;
>>    bitmap4           hintset;              /* Similar to
>>    opaque            hintlist<>;              fattr4 */
>> };
>>
>> struct pnfs_mdsthreshold {
>>    pnfs_mdsthreshold_item hints<>;
>> };
>>
>> Bitmap descriptions:
>>
>> Name            #  Data Type   Access   Description
>> ----            -  ---------   ------   -----------
>> size_threshold  1  length4     READ     The file size below which it is
>>                                          recommended to read/write data
>>                                          through the MDS
>>
>> Description
>> -----------
>>
>> This attribute acts as a hint to the client to help it determine when it 
> 
>> is more efficient to read/write a file to the metadata server vs. the 
>> dataserver.  Regardless of the structural representation, the meaning of 
> 
>> the size_threshold is the same.  Files with a size shorter than 
>> 'size_threshold' should be read/written to the MDS; all other files 
>> should be read/written to the data servers.
>>
>> As Benny pointed out in an earlier email:
>>
>> If the size_threshold is set to 0 then no files (shorter than or equal 
>> to 0 bytes) will be read from/written to the MDS, if it's set to ~0 then 
> 
>> all files (shorter than or equal to ~0 bytes) will be read from/written 
>> to the MDS.
>>
>> An array of hints based on layout type is used, as a server may support 
>> multiple layout types and may want to be able to return different 
>> attribute values depending on the layout type.  In the common case, this 
> 
>> will be a short list.
>>
>> A file or FSID attribute?
>> -------------------------
>> There have been arguments made both ways.  An FSID attribute allows a 
>> client to query it, before the file is open.  However, one could always 
>> query the attribute prior to performing an I/O.
>>
>> An alternate solution is to make the attribute available both as a FSID 
>> and a file attribute and allow the server to decide if files have 
>> different attribute values than the file system.  Thus, a query using a 
>> file's file handle may return the generic file system attribute or a 
>> specific attribute for only that file.  If the client wants a generic 
>> file system attribute it should not call the GETATTR with a file 
> filehandle.
>> Comments?
>>
>> Thanks,
>> -Garth
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4