Subject: Re: [nfsv4] Device stateids (v3)

From: Benny Halevy <[email-masked]>
Date: Mon, 05 Nov 2007 12:47:09 +0200

I updated and hopefully simplified the proposal based on Dave's
comments.

- device ID major/minor is used instead of devsetid.
- CB_DEVICERECALL merged into CB_DEVICENOTIFY (now used
  for deletions, changes, and additions)
- removed DEVICERETURN
- defined new SEQ4_STATUS_DEVICE_STATE_REVOKED sequence flag.

Benny
--
1. Introduction

The NFS client maintains a mapping of device IDs contained in a layout
to the corresponding storage device addresses. Device IDs are comprised
of { di_major, di_minor } allowing the server to partition the device
ID namespace along its file systems architecture boundaries (e.g., each
di_major can be associated with a filesystem).

The state of the mapping is represented by a recallable device stateid
(devstateid). There is at most one devstateid per { client ID, layout
type } nexus.

GETDEVICELIST and GETDEVICEINFO are used to retrieve the device ID
mapping information.  CB_DEVICENOTIFY is a callback used to notify
the client of changes to the device ID mapping.
SEQ4_STATUS_DEVICE_STATE_REVOKED is new sequence flag used to
notify the client the its device ID mapping state has been revoked
if the server failed to notify the client of changes to the device 
ID mapping via CB_DEVICENOTIFY.

2. Data structures

struct deviceid4 {
    uint64_t        di_major;
    uint64_t        di_minor;
};

3. Getting device ID mapping information

GETDEVICEINFO
SYNOPSIS: devstateid, layout_type, deviceid, maxcount ->
	devstateid, device_addr

GETDEVICEINFO returns device mapping information for a fully specified
deviceid. The server should return NFS4ERR_INVAL if deviceid.di_minor
is set to zero.

The current filehandle is not used.

GETDEVICELIST

struct gdl_devmajor4 {
    bool             gdldm_valid;
    uint64_t         gdldm_major;
};

SYNOPSIS: devstateid, layout_type, devmajor, maxcount, cookie,
          cookieverf ->
	devstateid, cookie, cookieverf, device info list<>

GETDEVICELIST returns a list of device ID mapping information. When
devmajor.gdldm_valid is false it lists all device IDs for the specified
layout_type, otherwise, GETDEVICELIST lists all the device IDs for the
specified layout_type where deviceid.di_major equals the
devmajor.gdldm_major. The latter method will typically be used
following a respective CB_DEVICENOTIFY for <devmajor, 0>

The current filehandle is not used.

GETDEVICEINFO and GETDEVICELIST each take the a devstateid and the
layout type and return a non-zero devstateid. In case the client holds
a valid devstateid it MUST use it for subsequent GETDEVICEINFO and
GETDEVICELIST that update the device ID mapping.  The devstateid
argument MUST be set to zero whenever the client has no valid
devstateid.

Note that LAYOUTGET and GETDEVICELIST MUST NOT return any deviceid
where deviceid.di_minor equals zero.

4. Notifying the client of changes to the device ID mapping

CB_DEVICENOTIFY

enum devicenotify_type4 {
    DEVICENOTIFY_DELETED    = 1,
    DEVICENOTIFY_CHANGED    = 2,
    DEVICENOTIFY_ADDED      = 3,
};

SYNOPSIS: devstateid, layout_type, notify_type, deviceid<> ->

CB_DEVICENOTIFY takes a devstateid, the layout_type, a notification
type, and an array of device IDs to notify the client of deletions,
updates, or additions to the device ID mapping.  When replying to
CB_DEVICENOTIFY, the client should invalidate the specified device IDs
such that any futher use of them will require the use of GETDEVICEINFO
or GETDEVICELIST to retrieve the updated mapping.  Note, however, that
some I/O operations to the affected storage devices may still be
outstanding.   The server MAY fence the client off from the changed
device IDs until the client updates the mapping yet it is not required
to do so. If the server needs to synchronize with the client with
that respect, it SHOULD recall all layouts referring to the affected
device IDs and wait for the client to return the layouts.

For each deviceid, a zero value for deviceid.di_minor represents all
device IDs matching deviceid.di_major.  Typically, in this case the
client will use GETDEVICELIST for deviceid = <di_major, 0> to update
all the affected device mappings.

Deleted device IDs will not be reused by the server and all outstanding
layouts referring to it are expected to be recalled via CB_LAYOUTRECALL;
Attempting to use GETDEVICEINFO for the deleted deviceids is allowed but
rather useless as the server MUST reject it with NFS4ERR_INVAL.

The server can inform the client of additional deviceids that were not
previously returned in a previous series of GETDEVICELIST calls (where
the series ended in gdlr_eof == TRUE). The purpose of this operation is
to reduce the latency for getting new device mappings.  Rather than
waiting until the client accesses a file that uses the new device ID,
issue a LAYOUTGET, and then a GETDEVICEINFO for the new device, the
server can notify the client of any new device ID in advance and have
the client get the device address mapping by calling GETDEVICEINFO or
GETDEVICELIST at its convenience.

For each added deviceid, a deviceid.di_minor set to zero is an
indication that a new set of devices sharing the specified di_major
were added and the client SHOULD use GETDEVICELIST to get the all
mappings in a batch operation, otherwise the client MAY use
GETDEVICEINFO for each specified deviceid.

If the client has already issued a GETDEVICEINFO for the added device
IDs, it MAY just ignore the device id notification.

CB_DEVICENOTIFY does not change the devstateid.

5. The SEQ4_STATUS_DEVICE_STATE_REVOKED SEQUENCE flag

In addition to informing the client of mapping changes via
CB_DEVICENOTIFY, the server MUST use the (newly introduced)
SEQ4_STATUS_DEVICE_STATE_REVOKED SEQUENCE flag to indicate that all
device ID mappings have been revoked without expiration of the lease
period due to the server failure to notify the client of changes in
the mapping via CB_DEVICENOTIFY. This status bit remains set on all
SEQUENCE replies until the loss of the device mapping state has been
acknowledged by use of either GETDEVICEINFO or GETDEVICELIST using the
zero devstateid.

On Oct. 31, 2007, 19:59 +0200, Benny Halevy <[email-masked]> wrote:
> Dave, thanks for the comments!
> My answers inline below.
> 
> Benny
> 
> On Oct. 31, 2007, 2:24 +0200, "Noveck, Dave" <[email-masked]> wrote:
>>> Plus, I simplified the proposal by not requiring any new sequence
>>> flags as the server should use the proposed callback operation to
>>> recall device mappings and use the
>> SEQ4_STATUS_RECALLABLE_STATE_REVOKED 
>>> and SEQ4_STATUS_CB_PATH_DOWN sequence flags as appropriate the 
>>> same it may use them for other recallable state objects (locks and 
>>> layouts).
>> As far as I can see you don't you them the same and that is an issue. 
> 
> OK. we can define SEQ4_STATUS_DEVID_DELETED* in the spirit of the
> last proposal [that I see is now part of draft-15 which seems to
> introduce inconsistencies with the side effects of LAYOUTRETURN
> for RETURN_{FSID,ALL} that return the respective device mappings
> but do not change the device stateid]
> 
>>> The state of the mapping is represented by a recallable device 
>>> stateid (devstateid). There is at most one devstateid per { client ID,
>>> layout type, devsetid } nexus.
>>> A recommended filesystem attribute - devsetid is introduced.
>> Notice that this arrangement means that fsid A and fsid B share a 
>> devsetid or layout type X iff they share it for layout type Y.
>> It seems that layout type is the one that should have the flexibility
>> to figure out how it arranges its devices.
>>
>> I think a better model is to divide device id into major and minor
>> fields and then all devices sharing a major device value become a device
>> set.  I think this would give you a lot more flexibility.
> 
> Yeah, I see your point. Good idea.
> 
>>> DEVICERETURN returns a device ID mapping.  When called, this client 
>>> MUST guarantee not to make further use of that device ID mapping.
>> Do you really mean that?  Isn't this like a CLOSE if which if you used 
>> the stateid for the open file, you would get an error.  Do you really
>> mean that he server can depend on you not using this?
> 
> same as LAYOUTRETURN. the server is not required to fence off the client after
> LAYOUTRETURN.
> 
>>> If the deviceid argument is set to zero, all device ID mappings in the
>>> referred devset are returned.  The server updates the devstateid and 
>>> returns the updated value.
>> If the deviceid were zero, the returned value of devstateid would be
>> zero.
>> Correct?
> 
> I'm not sure.  What does the server return on CLOSE?
> 
>>> GETDEVICEINFO
>>> SYNOPSIS: devstateid, devsetid, deviceid, layout_type, maxcount,
>> notify_ask ->
>>> 	devstateid, notify_ack, device_addr
>>> GETDEVICELIST
>>> SYNOPSIS: devstateid, devsetid, layout_type, maxcount, cookie,
>> cookieverf, notify_ask ->>
>>> 	devstateid, notify_ack, cookie, cookieverf, device info list<>
>>> GETDEVICEINFO and GETDEVICELIST each take the a devstateid, a
>> devsetid, 
>>> and the layout type, and return a non-zero devstateid.
>> This looks inconsistent to me.
>>
>> If I do a GETDEVICEINFO(a), I get a non-zero stateid.
>>
>> If I then do a DEVICERETURN(a), I get a zero stateid.
>>
>> OK, now I start with a zero stateid again and do a GETDEVICELIST and
>> get a non-zero stateid.
>>
>> If I do a GETDEVICELIST, I get another non-zero stateid.
>>
>> If I then do a DEVICERETURN(a), I get ... a non-zero stateid because
>> the GETDEVICELIST has some effect.  Is that true.
> 
> if you didn't return all device IDs using deviceid==0 then you
> shouldn't get a zero devstateid.
> 
>> It seems the DEVICERETURN is the inverse GETDEVICEINFO but that 
>> GETDEVICELIST has no inverse.  At the very least we need more clarity
>> about this stuff.
> 
> The way I see it DEVICERETURN w/ deviceid==0 is sort of the inverse
> of GETDEVICELIST as it "resets" (or reinitializes the device state.
> 
>>
>>> GETDEVICEINFO and GETDEVICELIST also take a boolean flag optionally 
>>> asking for device notifications and return a corresponding flag 
>>> confirming the notification.
>> Isn't this new function?  Is it really needed?
> 
> It was in the previous proposal.
> It is not strictly needed as the new device ID can always be introduced
> at LAYOUTGET time and be resolved using GETDEVICEINFO.
> The motivation is to optimize this case and reduce the latency by
> allowing the client to get the device info in advance (it could do
> so opportunistically, only if it's idle).
> 
>>> Device stateids follow the sequencing and use rules of all other
>> stateid 
>>> types. 
>> I assume that that means I use the normal "other" value with a seqid of
>> zero as a wildcard.
>>  
>>> The devstateid is invalidated when all devices associated with 
>>> it are returned via DEVICERETURN. The client can then "forget" about
>> that 
>>> devstateid and use a value of zero for the devstateid for any
>> subsequent 
>>> operation that takes a devstateid argument and allows the
>> uninitialized 
>>> value.
>> I suppose but the server can return that devstateid as zero.  Then the 
>> client can either remember that zero and use it in which case he is
>> using 
>> the correct thing or he can specially note it is zero and this allows
>> him
>> to forget it (but then he has to specially note that he has forgotten it
>> and should use zero in its stead).  Seems like the code would be simpler
>> if it just remembered the last thing it got back (if the client did
>> parallel once it would have to compare state seqids and treat zero
>> specially
>> as later than all normal ones.
> 
> Agreed. That makes a lot of sense.
> 
>>> 5. Recalling device IDs
>>> CB_DEVICERECALL
>>> SYNOPSIS: devstateid, devsetid, deviceid, deleted ->
>>>
>>> CB_DEVICERECALL takes a devstateid, devsetid, and a deviceid to notify
>>> the client it needs to return the specified deviceid (when non-zero)
>> or 
>>> all device IDs in the devsetid when the deviceid is set to zero, by 
>>> using a respective DEVICERETURN. CB_DEVICERECALL does not change the 
>>> device stateid.
>> A couple things are unclear.  It says it needs to return the specified
>> device id, but is that true when 'deleted' is false?  Also, what happens
>> when the device id is zero and 'deleted' is false.  What is supposed to
>> happen?
> 
> I meant that the client should send a DEVICERETURN in both cases,
> although it doesn't seem harmful if it issues a GETDEVICEINFO to
> refresh the data when "deleted" is set to false.  But the server will
> need to detect that case.
> 
>>> All layouts using the affected device IDs remain in force, but the 
>>> server SHOULD fence the client from accessing the affected devices 
>>> until the client has returned the recalled device IDs.
>> And after it has returned it?  Should the server not fence the client
>> form a device it has returned.
> 
> I don't think so.  We have access control mechanism to limit what
> the client can do on the device but fencing the client off may hurt
> performance needlessly.
> 
>>> When the "deleted" boolean flag is set to false the client MAY use 
>>> GETDEVICEINFO or GETDEVICELIST to get the updated device mapping if 
>>> it has valid layouts the refer to the changed deviceid.  
>> After or instead of doing the return?
> 
> See answer above.  This needs to be clarified and I think I'll go
> with the permissive approach of allowing to get the info without
> and explicit DEVICERETURN.
> 
>>> If the "deleted" flag is set to true the specified deviceid will not
>> be
>>> reused by the server and all outstanding layouts referring to it are 
>>> expected to be recalled via CB_LAYOUTRECALL; hence the client SHOULD 
>>> NOT attempt to call GETDEVICEINFO or GETDEVICELIST for the specified 
>>> deviceid. However, if it does do so the server MUST return a
>> NFS4ERR_INVAL 
>>> error.
>>> 6. Device notification
>>> CB_DEVICENOTIFY
>>> SYNOPSIS: devstateid, devsetid, deviceid ->
>>> If the server supports CB_DEVICENOTIFY for devices and the client has 
>>> requested device id notifications via GETDEVICEINFO or GETDEVICELIST 
>>> the server can inform the client of additional deviceids that were 
>>> not previously returned by GETDEVICELIST.
>> Again, what is the need here?  The client will see the device id's
>> when he gets a layout.
> 
> Agreed.  This is an optimization we can live without of.
> see above...
> 
>>> 7. SEQUENCE flags
>>> In addition to informing the client of mapping changes via 
>>> CB_DEVICERECALL and CB_DEVICENOTIFY, the server MUST use the 
>>> SEQ4_STATUS_RECALLABLE_STATE_REVOKED SEQUENCE flag to indicate 
>>> that one or more device IDs have been revoked without expiration 
>>> of the lease period, due to the client's failure to return them 
>>> when recalled. 
>> First of all, in the case of CB_DEVICENOTIFY, nothing has been 
>> recalled or is supposed to be returned, as far as I can see.
> 
> right.
> 
>> Then it speaks of devices being "revoked".  Devices do not have
>> stateids and are not, in that sense, locking objects.  If the 
>> current protocol, when a stateid has multiple locks (.e.g. byte-range
>> locks), it is carefully stateid that when one is revoked, all are as
>> a unit and that is tied to the associated stateid being made invalid.
>>
>> When SEQ4_STATUS_RECALLABLE_STATE_REVOKED is set in the current 
>> protocol, the client is supposed to use TEST_STATEID to determine
>> what state has been lost and acknowledge that loss using FREE_STATEID.
>>
>> I think it would be much better if revocation continued to follow
>> that model.  If a devstateid is revoked, then, at least for that,
>> you start from scratch. 
> 
> OK
> 
>>> This status bit remains set on all SEQUENCE replies until the 
>>> loss of all such device ID mappings has been acknowledged by use 
>>> of DEVICERETURN.
>> I think it should stay until all revoked stateids are freed via
>> FREE_STATEID, just as it is today.
> 
> OK
> 
>>
>> -----Original Message-----
>> From: Benny Halevy [mailto:[email-masked]] 
>> Sent: Monday, October 29, 2007 11:45 AM
>> To: Robert Gordon; Marc Eshel
>> Cc: NFSv4
>> Subject: Re: [nfsv4] Device stateids
>>
>> During the most recent bakeathon concern has been raised about the
>> global device ID namespace and Marc Eshel from IBM requested that we
>> retain the ability to recall devices per-fsid.  To satisfy this
>> requirement as well as the requirement for a simple global device ID
>> namespace the updated proposal below introduces a device-set identifier
>> (devsetid) that qualifies the device ID.
>>
>> This provides for a more flexible deviceid namespace topology that can
>> cover either a global deviceid namespace, per-fsid namespace, or
>> anything in between (e.g. when several filesystems reside on (and are
>> confined to) a set of devices, like in EMC or Panasas' cases).
>>
>> The proposal also introduces new operations to manage device mapping
>> state rather than unnecessarily overloading existing operations.
>>
>> Plus, I simplified the proposal by not requiring any new sequence flags
>> as the server should use the proposed callback operation to recall
>> device mappings and use the SEQ4_STATUS_RECALLABLE_STATE_REVOKED and
>> SEQ4_STATUS_CB_PATH_DOWN sequence flags as appropriate the same it may
>> use them for other recallable state objects (locks and layouts).
>>
>> One last change is providing for a "wildcard" value (0) for device ID
>> that specifies all the device IDs in a devset.  This means that the
>> server MUST NOT return a zero device ID in GETDEVICELIST or in any
>> layout and it must reject it as invalid in GETDEVICEINFO.GETDEVICEINFO
>> and GETDEVICELIST also take a boolean flag optionally asking for device
>> notifications and return a corresponding flag confirming the
>> notification.
>> I guess that this won't be an issue for anyone; otherwise, please
>> holler.
>>
>> Benny
>>
>> --
>> 1. Introduction
>>
>> The NFS client maintains a mapping of device IDs contained in a layout
>> to the corresponding storage device address.
>> Device IDs and filesystem IDs are associated with device set IDs such
>> that each fsid is associated with a single devsetid as indicated by a
>> new, recommended filesystem attribute, and each deviceid is associated
>> with a single devsetid.
>>
>> The state of the mapping is represented by a recallable device stateid
>> (devstateid). There is at most one devstateid per { client ID, layout
>> type, devsetid } nexus.
>>
>> A recommended filesystem attribute - devsetid is introduced.
>> All deviceids associated with any file in the filesystem are associated
>> with this devsetid. 
>> If the server does not support device sets, it can omit this attribute,
>> making the deviceid namespace global and implicitly associating all
>> deviceids with the default devset (having the implicit devsetid ID of 0)
>>
>> 2. Getting device information
>>
>> GETDEVICEINFO
>> SYNOPSIS: devstateid, devsetid, deviceid, layout_type, maxcount,
>> notify_ask ->
>> 	devstateid, notify_ack, device_addr
>>
>> GETDEVICELIST
>> SYNOPSIS: devstateid, devsetid, layout_type, maxcount, cookie,
>> cookieverf, notify_ask ->
>> 	devstateid, notify_ack, cookie, cookieverf, device info list<>
>>
>> GETDEVICEINFO and GETDEVICELIST each take the a devstateid, a devsetid,
>> and the layout type, and return a non-zero devstateid.
>> In case the client holds a valid devstateid it MUST use it for
>> subsequent GETDEVICEINFO and GETDEVICELIST that update the device ID
>> mappings.  The devstateid argument must be set to zero whenever the
>> client has no valid devstateid (e.g. after reboot or after the
>> devstateid was previously returned via DEVICERETURN).
>>
>> GETDEVICEINFO and GETDEVICELIST also take a boolean flag optionally
>> asking for device notifications and return a corresponding flag
>> confirming the notification.
>>
>> 3. Returning device IDs
>>
>> DEVICERETURN
>> SYNOPSIS: devstateid, devsetid, deviceid -> devstateid
>>
>> DEVICERETURN returns a device ID mapping.  When called, this client MUST
>> guarantee not to make further use of that device ID mapping.
>> If the deviceid argument is set to zero, all device ID mappings in the
>> referred devset are returned.  The server updates the devstateid and
>> returns the updated value.
>>
>> 4. Sequencing rules
>>
>> Device stateids follow the sequencing and use rules of all other stateid
>> types.  The devstateid is invalidated when all devices associated with
>> it are returned via DEVICERETURN. The client can then "forget" about
>> that devstateid and use a value of zero for the devstateid for any
>> subsequent operation that takes a devstateid argument and allows the
>> uninitialized value.
>>
>> 5. Recalling device IDs
>>
>> CB_DEVICERECALL
>> SYNOPSIS: devstateid, devsetid, deviceid, deleted ->
>>
>> CB_DEVICERECALL takes a devstateid, devsetid, and a deviceid to notify
>> the client it needs to return the specified deviceid (when non-zero) or
>> all device IDs in the devsetid when the deviceid is set to zero, by
>> using a respective DEVICERETURN. CB_DEVICERECALL does not change the
>> device stateid.
>>
>> All layouts using the affected device IDs remain in force, but the
>> server SHOULD fence the client from accessing the affected devices until
>> the client has returned the recalled device IDs.
>>
>> When the "deleted" boolean flag is set to false the client MAY use
>> GETDEVICEINFO or GETDEVICELIST to get the updated device mapping if it
>> has valid layouts the refer to the changed deviceid.  If the "deleted"
>> flag is set to true the specified deviceid will not be reused by the
>> server and all outstanding layouts referring to it are expected to be
>> recalled via CB_LAYOUTRECALL; hence the client SHOULD NOT attempt to
>> call GETDEVICEINFO or GETDEVICELIST for the specified deviceid. However,
>> if it does do so the server MUST return a NFS4ERR_INVAL error.
>>
>> 6. Device notification
>>
>> CB_DEVICENOTIFY
>> SYNOPSIS: devstateid, devsetid, deviceid ->
>>
>> If the server supports CB_DEVICENOTIFY for devices and the client has
>> requested device id notifications via GETDEVICEINFO or GETDEVICELIST the
>> server can inform the client of additional deviceids that were not
>> previously returned by GETDEVICELIST.
>>
>> The purpose of this operation is to reduce the latency for getting new
>> device mappings.  Rather than waiting until the client accesses a file
>> that uses the new device ID, issue a LAYOUTGET, and then a GETDEVICEINFO
>> on the new device, the server can notify the client of any new device ID
>> in advance and have the client get the device address mapping by calling
>> GETDEVICEINFO at its convenience.
>>
>> CB_DEVICENOTIFY takes a device stateid, a device set ID and a device ID
>> in the device set for a device ID that was not returned in a previous
>> series GETDEVICELIST calls (where the series ended in gdlr_eof == TRUE).
>> The client calls GETDEVICEINFO or GETDEVICELIST to determine the
>> mappings for the new device ID.
>> CB_DEVICENOTIFY does not change the device stateid.
>>
>> If the client has already issued a GETDEVICEINFO for the specified
>> deviceid then it SHOULD just ignore the device id notification.
>> Updating an existing device ID mapping should be done using a sequence
>> of CB_DEVICERECALL, DEVICERETURN, and GETDEVICEINFO (or GETDEVICELIST).
>>
>> 7. SEQUENCE flags
>>
>> In addition to informing the client of mapping changes via
>> CB_DEVICERECALL and CB_DEVICENOTIFY, the server MUST use the
>> SEQ4_STATUS_RECALLABLE_STATE_REVOKED SEQUENCE flag to indicate that one
>> or more device IDs have been revoked without expiration of the lease
>> period, due to the client's failure to return them when recalled. This
>> status bit remains set on all SEQUENCE replies until the loss of all
>> such device ID mappings has been acknowledged by use of DEVICERETURN.
>>
>> On Aug. 09, 2007, 10:22 +0300, Benny Halevy <[email-masked]> wrote:
>>> Robert Gordon wrote:
>>>> In general i think this is good.
>>>>
>>>> I've started to think that the SEQUENCE status flags could just be 
>>>> two bits.
>>>>
>>>> SEQ4_STATUS_DEVID_DELETED_ALL
>>>> 	This flag stays on until the client issues a DELEGRETURN
>>>> 	using the devstateid.
>>>>
>>>> SEQ_STATUS_DEVID_CHANGED
>>>> 	This flag stay on until a GETDEVICELIST or GETDEVICEINFO
>>>> 	(if using CB_NOTIFY) gets the affected mappings.
>>> The DEVID_DELETED cases should be covered by CHANGED as the client 
>>> will have to revalidate all the device mappings anyway to see what 
>>> actually changed and should either not see the deleted deviceid 
>>> returned by GETDEVICELIST or get an error from GETDEVICEINFO for that
>> deviceid.
>>> The DEVID_ADDED case is covered if the client is doing GETDEVICELIST 
>>> and the server supports it.  If the server does not support 
>>> GETDEVICELIST is must not set SEQ_STATUS_DEVID_CHANGED for added 
>>> devices as the client has no way to get the new device ID.  These will
>>> be discovered only when the client will get new layouts referring to 
>>> the new devices (which is perfectly fine)
>>>
>>>> Robert.
>>>>
>>>> On Aug 8, 2007, at 10:58 AM, Benny Halevy wrote:
>>>>
>>>>> The NFS client maintains a mapping of device ids contained in a 
>>>>> layout, to the corresponding storage device address.
>>>>> The state of the mapping is represented by a single recallable 
>>>>> device stateid (devstateid). There is at most one devstateid per { 
>>>>> client ID, layout type } pair.
>>>>>
>>>>> GETDEVICEINFO and GETDEVICELIST each take the layout type and an 
>>>>> optional devstateid and return the devstateid.  In case the client 
>>>>> holds a valid devstateid it MUST use it for subsequent GETDEVICEINFO
>>>>> and GETDEVICELIST that update the device ID mappings.  The 
>>>>> devstateid argument is invalid whenever the client has no valid 
>>>>> devstateid (e.g. after reboot or after the devstateid was previously
>>>>> returned via DELEGRETURN).
>>>>> GETDEVICEINFO or GETDEVICELIST that operate on a device stateid that
>>>>> was already recalled via CB_RECALL MUST be rejected by the server 
>>>>> with NFS4ERR_BADSTATEID.
>>>>>
>>>>> GETDEVICEINFO and GETDEVICELIST also take a boolean flag optionally 
>>>>> asking for device notifications and return a corresponding flag 
>>>>> confirming the notification.
>>>>>
>>>>> Device stateids follow the sequencing and use rules of all other 
>>>>> stateid types.  The mappings are invalidated when the devstateid is 
>>>>> returned via CB_DELEGRETURN or explicitly deleted via CB_NOTIFY.
>>>>>
>>>>> All layouts using the device IDs remain in force, but the server 
>>>>> MUST fence the client from accessing the affected devices until the 
>>>>> client has obtained the re-mapped device IDs via GETDEVICEINFO or 
>>>>> GETDEVICELIST.
>>>>>
>>>>> CB_RECALL takes a devstateid to notify the client it needs to return
>>>>> all device IDs identified by the devstateid.  The filehandle 
>>>>> argument has no use in this case.  It does not change the device 
>>>>> stateid.
>>>>>
>>>>> DELEGRETURN takes a devstateid to return all device IDs associated 
>>>>> with the devstateid. The current filehandle has no use in this case.
>>>>>
>>>>> If the server supports CB_NOTIFY for devices and the Client has 
>>>>> requested device id notifications via GETDEVICEINFO or 
>>>>> GETDEVICELIST; The server can inform the client of a delete, add, or
>>>>> change mapping.
>>>>>
>>>>> CB_NOTIFY takes a device stateid qualifying a device ID for the 
>>>>> following device notifications:
>>>>>
>>>>> NOTIFY4_DELETE_DEVICE_ID
>>>>> 	Deletes all mappings using the specified device ID.
>>>>> 	The server MUST stop using the device ID. Any
>>>>> 	layouts remain in force, but aren't useful until
>>>>> 	new mappings are available. The server fences the
>>>>> 	client from the affected data server until all
>>>>> 	layouts with the device ID are returned, or the
>>>>> 	client obtains new mappings. The server provides
>>>>> 	a hint as to whether new mappings will arrive and
>>>>> 	if so how long.
>>>>>
>>>>> NOTIFY4_ADD_DEVICE_ID
>>>>> 	Adds a device ID to address mapping. This is
>>>>> 	for a device ID the that was not returned in a
>>>>> 	previous series GETDEVICELIST calls (where the
>>>>> 	series ended in gdlr_eof == TRUE).  The client
>>>>> 	calls GETDEVICEINFO or GETDEVICELIST to determine
>>>>> 	the mappings for the new device ID.
>>>>>
>>>>> NOTIFY4_CHANGE_DEVICE_ID
>>>>> 	The client call GETDEVICEINFO or GETDEVICELIST
>>>>> 	to get updated device addresses for the device
>>>>> 	ID. Layouts remain in force. The client is fenced
>>>>> 	from any devices that were in the old address list,
>>>>> 	but not the new address list.
>>>>>
>>>>> CB_NOTIFY does not change the device stateid.
>>>>>
>>>>> In addition to informing the client of mapping changes via CB_NOTIFY
>>>>> and CB_RECALL, SEQUENCE carries status flags to indicate that device
>>>>> mappings have changed.
>>>>>
>>>>> The flags are :-
>>>>>
>>>>> SEQ4_STATUS_DEVID_DELETED
>>>>> 	Mapping has been partially deleted. If the client is
>>>>> 	using CB_NOTIFY it may issue a GETDEVICEINFO for each
>>>>> 	notified deletion or GETDEVICELIST to refresh the entire
>>>>> 	map. This flag stays on until the client responds to
>>>>> 	any and all _DELETE notifications, or it has issued a
>>>>> 	GETDEVICELIST.
>>>>>
>>>>> SEQ4_STATUS_DEVID_DELETED_ALL
>>>>> 	This flag stays on until the client issues a DELEGRETURN
>>>>> 	using the devstateid.
>>>>>
>>>>> SEQ4_STATUS_DEVID_CHANGED
>>>>> SEQ4_STATUS_DEVID_ADDED
>>>>> 	These flags stay on until a GETDEVICELIST or GETDEVICEINFO
>>>>> 	(if using CB_NOTIFY) gets the affected mappings.
>>>>>
>>>> _______________________________________________
>>>> nfsv4 mailing list
>>>> [email-masked]
>>>> https://www1.ietf.org/mailman/listinfo/nfsv4
>>> _______________________________________________
>>> nfsv4 mailing list
>>> [email-masked]
>>> https://www1.ietf.org/mailman/listinfo/nfsv4
>>
>> _______________________________________________
>> nfsv4 mailing list
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/nfsv4
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Robert Gordon <[email-masked]>
Date: Mon, 5 Nov 2007 12:33:43 -0600

> - defined new SEQ4_STATUS_DEVICE_STATE_REVOKED sequence flag.

So, in this proposal are you suggesting that CB_DEVICENOTIFY
is a mandatory operation to implement at the client ?

also, i had thought that the deviceid moving from 32 bits to 64
was to accommodate maj/minor partitioning; now we have a 128 bit
deviceid ?? is that really necessary ?

Robert.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 05 Nov 2007 20:55:35 +0200

> is a mandatory operation to implement at the client ?

yes, pretty much as CB_DEVICERECALL was in the last proposal.
The client can ignore the ADDED notifications though if it wants.

> deviceid ?? is that really necessary ?

not exactly.  The move to 64 bit was to accommodate for the existing
64-bit device IDs we have implemented in panfs.  The 64-bit major
is to match fsid.major.  This shouldn't be a big deal for files
as you have very few device IDs and only one in each layout.
For objects I thought of changing the pnfs-obj layout to have one major
for the layout and just minor for each device in the map but I'm
really not sure it's worth it.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 5 Nov 2007 16:39:17 -0500

> The client can ignore the ADDED notifications though if it wants.

pNFS is not a mandatory feature so I assume that a client can
return NFS4ERR_NOTSUPP.  Are we saying you MUST NOT do that if
you have done one of the pNFS operations?

---

From: Benny Halevy <[email-masked]>
Date: Mon, 05 Nov 2007 23:57:04 +0200

> you have done one of the pNFS operations?

This will make sense. and the same for CB_LAYOUTRECALL.

for a MDS supporting pNFS, I'd say that the following operations must be implemented:
 GETDEVICEINFO, (GETDEVICELIST can remain optional)
 LAYOUTCOMMIT,
 LAYOUTGET,
 LAYOUTRETURN

for a client that supports pNFS and issued one of the pNFS operations:
 CB_DEVICENOTIFY,
 CB_LAYOUTRECALL (if issued LAYOUTGET)

---

From: Mike Eisler <[email-masked]>
Date: Tue, 6 Nov 2007 02:41:20 -0800 (PST)

>  CB_LAYOUTRECALL (if issued LAYOUTGET)

Why can't CB_NOTIFY be used in instead of CB_DEVICENOTIFY?

---

From: Benny Halevy <[email-masked]>
Date: Tue, 06 Nov 2007 14:04:45 +0200

> Why can't CB_NOTIFY be used in instead of CB_DEVICENOTIFY?
Well, it can, but I think overloading a callback meant for directory 
notifications
doesn't make the the client's life any easier. The fact that device 
mapping resemble
directory delegations by the fact that entries can be added, changed, or 
deleted
from the state is insufficient IMO since the actual notification contents
are totally different.

Having a separate callback operation for device notifications rather 
than doing both
device and directory notification using the same callback is simpler to 
understand and to
implement. since the respective xdr routine and the upper layer service 
wouldn't have
to fork on the notification type to see if they're dealing with a 
directory or with device IDs.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Tue, 6 Nov 2007 04:44:27 -0800 (PST)

> Well, it can, but I think overloading a callback meant for directory 

And in draft-15, it does.

This change was made in draft-15 based on a perceived rough
consensus reach months ago. I've read reports of complaints
during last months bake-a-thon, but those complaints were never
aired on the WG list or in an official WG meeting.

There isn't great deal of time left to make edits and we've got
substantive changes to make. Unfortunately, adding a new op
is something that we don't always get right the first time.

> from the state is insufficient IMO since the actual notification

Which is analogous to saying that because ACLs are totally different
from file size, two separate operations should be used for retrieving
ACLs and file sizes. But instead we have a bitmask for GETATTR to
allow clients to selectively get attributes they are interested in.
Similarly CB_NOTIFY uses a bitmap to selectively push events.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 06 Nov 2007 16:44:00 +0200

> aired on the WG list or in an official WG meeting.
I'm not sure that the perception of rough consensus was common to everybody.
That's probably an indication that we all need to communicate our thoughts
and intentions more clearly on the mailing list.
> is something that we don't always get right the first time.
I apologize for not raising this concern earlier.
It's not a big deal so if you feel more comfortable with doing both in 
CB_NOTIFY
then it can be done.
> Similarly CB_NOTIFY uses a bitmap to selectively push events.
[This can be developed further on a separate thread :) but actually I 
think that the file size
should indeed be handled differently as setting it is a data modifying 
operation...]

Speaking of the bitmap in CB_NOTIFY wouldn't it be simpler if instead of 
having
a list of  struct notify4, each having a bitmap and an opaque array 
we'll have a
list of discriminated unions (especially as the draft says "The 
notify_type4 can only have one
bit set for each notification in the array")

for example:

enum notify_type4 {
        NOTIFY4_CHANGE_CHILD_ATTRS = 0,
        NOTIFY4_CHANGE_DIR_ATTRS = 1,
        NOTIFY4_REMOVE_ENTRY = 2,
        NOTIFY4_ADD_ENTRY = 3,
        NOTIFY4_RENAME_ENTRY = 4,
        NOTIFY4_CHANGE_COOKIE_VERIFIER = 5,
        NOTIFY4_DEVICE_ID_ADD = 6,
        NOTIFY4_DEVICE_ID_CHANGE = 7,
        NOTIFY4_DEVICE_ID_DELETE = 8
};

union notify4 switch (notify_type4 notify_type) {
   case NOTIFY4_CHANGE_CHILD_ATTRS:
        notify_attr4        notify_child_attr;

   case NOTIFY4_CHANGE_DIR_ATTRS:
        notify_attr4        notify_dir_attr;
	/* or just fattr4 for holding the new attrs as ne_file isn't needed in this case? */

   case NOTIFY4_REMOVE_ENTRY:
        notify_remove4      notify_remove;

   case NOTIFY4_ADD_ENTRY:
        notify_add4         notify_add;

   case NOTIFY4_RENAME_ENTRY:
        notify_rename4      notify_rename;

   case NOTIFY4_CHANGE_COOKIE_VERIFIER:
        notify_verifier4    notify_verifier;

   case NOTIFY4_DEVICE_ID_ADD:
   case NOTIFY4_DEVICE_ID_CHANGE:
   case NOTIFY4_DEVICE_ID_DELETE:
        notify_deviceid4    notify_deviceid;
}

struct CB_NOTIFY4args {
        stateid4    cna_stateid;
        nfs_fh4     cna_fh;
        notify4     cna_changes<>;
};

---

From: Robert Gordon <[email-masked]>
Date: Tue, 06 Nov 2007 10:09:12 -0600

>  GETDEVICEINFO, (GETDEVICELIST can remain optional)

I'm generating a table that lists the mandatory/optional/mandatory  
not to implement
disposition for each operation.

I was surprised to see benny claim that GETDEVICELIST is optional;  
Since no
one has objected .. then I'll assume it IS optional.

--
Robert...
"Don't Make Assumptions"

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 6 Nov 2007 11:17:29 -0500

pnfs is an optional feature so all the ops associated with have to
be optional.  Maybe for things like this that are mandatory if you
implement a particular feature but the feature is optional, the 
table should give the name of the particular feature, rather than
simply saying "optional".

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Tuesday, November 06, 2007 11:09 AM
To: NFSv4
Subject: Re: [nfsv4] Device stateids (v3)

--
Robert...
"Don't Make Assumptions"

---

From: Robert Gordon <[email-masked]>
Date: Tue, 06 Nov 2007 10:26:06 -0600

> simply saying "optional".

I agree, the table will reflect this.

I think that benny was saying that if you implement the
optional feature pnfs, the GETDEVICELIST operation would
be optional to implement.

--
Robert...
"Don't Make Assumptions"

---

From: Marc Eshel <[email-masked]>
Date: Tue, 6 Nov 2007 09:06:17 -0800

It is not enough to say that GETDEVICELIST can return an empty list an any 
server that don't want to implement it just return an empty list?
Marc.

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 06 Nov 2007 11:50:58 -0600

> simply saying "optional".

Yes, it should.  LAYOUTGET wouldn't be very helpful unless
LAYOUTRETURN were also implemented.

Without further justification to the contrary, I believe that
GETDEVICELIST should be mandatory to implement if the pNFS feature is
supported.

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 06 Nov 2007 11:52:11 -0600

> server that don't want to implement it just return an empty list?

No.  If the server is returning an empty list than that means
something very different than the operation is not supported.
It is misleading to the client even if the words in the specification
were to say something to the contrary

Spencer

---

From: Marc Eshel <[email-masked]>
Date: Tue, 6 Nov 2007 09:58:23 -0800

> were to say something to the contrary

How does it help the client to get not supported vs. empty list ? what can 
the client do differently based on the 2 different responses ?
Marc.

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 06 Nov 2007 12:06:20 -0600

> the client do differently based on the 2 different responses ?

an operation level error of ENOTSUPP is very clear where an empty list
may actually be a valid response where the server may support
the GETDEVICELIST operation but just not have any deviceids available
at that point in time.

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Tue, 06 Nov 2007 20:08:10 +0200

> be optional to implement.

Correct.
It is an optimization.  pNFS should work flawlessly just
by using GETDEVICEINFO on-demand whenever the client needs
to resolve a device ID that it hasn't seen before or that
fell out of its cache.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Tue, 06 Nov 2007 20:10:00 +0200

> were to say something to the contrary

Agreed.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Tue, 06 Nov 2007 20:36:29 +0200

> GETDEVICELIST should be mandatory to implement if the pNFS feature is

why?
Our server has no idea of a device list at the moment but it knows
to get the device info for each device Id it returns in a layout.
This must be sufficient for the client.

Benny

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 06 Nov 2007 12:51:10 -0600

> This must be sufficient for the client.

You have provided the justification.  Your server will refuse to
or is incapable of implementing a mechanism to support device lists.

Spencer