Subject: [nfsv4] Proposal for <cref> relative to MDS COMMIT

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 8 Nov 2007 15:11:18 -0500

I'm proposing replacing the following,

<t>
 The flag NFL4_UFLG_COMMIT_THRU_MDS in the field nfl_util of the file
layout
 (data type nfsv4_1_file_layout4) or the field nflh_util of
 the layout hint (data type nfsv4_1_file_layouthint4) is an indication
from
 the metadata server to the client of the preferred way of
 performing COMMIT. If this field is TRUE, the client
 SHOULD send COMMIT to the metadata server instead of
 sending it to the same data server to which the associated
 WRITEs were sent. If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is TRUE,
 then in order to maintain the current NFSv4.1 commit and
 recovery model, the data servers MUST return a common
 writeverf verifier in all WRITE responses for a given file
 layout, and the metadata server's COMMIT implementation
 must return the same writeverf.  The value of the
 writeverf verifier MUST be changed at the metadata server
 or any data server that is referenced in the layout,
 whenever there is a server event that can possibly lead to
 loss of uncommitted data.  The scope of the verifier can
 be for a file or for the entire pNFS server.  It might be
 more difficult for the server to maintain the verifier
 at the file level but the benefit is that only events
 that impact a given file will require recovery action.
 <cref>
 Trond comments:
 The whole section needs justification: there is no discussion anywhere
 of why a server implementor might prefer to receive COMMIT requests
through
 the MDS. Why couldn't such a server simply perform an implicit COMMIT
 on LAYOUTCOMMIT since the latter has persistence guarantees anyway on
 the metadata?
 Marc Eshel to propose text.
 </cref>The justification for this arises from implementation
considerations.  As 
 long as the justification is sound, I don't think that is an issue.
 </t>

with:

 <t>
   The file layout provides two alternate means of providing for the 
   commit of data written through data servers.  The flag 
   NFL4_UFLG_COMMIT_THRU_MDS in the field nfl_util of the file layout
   (data type nfsv4_1_file_layout4) or the field nflh_util of
   the layout hint (data type nfsv4_1_file_layouthint4) is an indication

   from the metadata server to the client of the preferred way of
   performing COMMIT, either by sending the COMMIT to the data server
   or the metadata server.  These two methods of dealing with the issue 
   correspond to broad styles of implementation for a pNFS server
supporting
   the files layout type.
   <list style="symbols">
     <t>
       When the flag is false, COMMIT operations are to be done to the
data
       server to which the corresponding writes were done.  This
approach
       is most useful when striping of files is implemented as part of
       pNFS server, with the individual data servers each implementing 
       their own file systems.
     </t>
     <t>
       When the flag is true, COMMIT operations are done to the metadata
       server, rather than to the individual data servers.  This
approach 
       is most useful when the pNFS server is implemented on top of a 
       clustered file system.  In such an implementation, sending
COMMIT's
       to multiple data servers may result in repeated writes of
metadata
       blocks as each individual COMMIT is executed, to the detriment of
       write performance.  Sending a single COMMIT to the metadata
server
       can provide more efficiency when there exists a clustered file 
       system capable of implementing such a co-ordinated COMMIT.
     <vspace blanklines='1' />
       If nfl_util & NFL4_UFLG_COMMIT_THRU_MDS is TRUE,
       then in order to maintain the current NFSv4.1 commit and
       recovery model, the data servers MUST return a common
       writeverf verifier in all WRITE responses for a given file
       layout, and the metadata server's COMMIT implementation
       must return the same writeverf.  The value of the
       writeverf verifier MUST be changed at the metadata server
       or any data server that is referenced in the layout,
       whenever there is a server event that can possibly lead to
       loss of uncommitted data.  The scope of the verifier can
       be for a file or for the entire pNFS server.  It might be
       more difficult for the server to maintain the verifier
       at the file level but the benefit is that only events
       that impact a given file will require recovery action.   
     </list>
   </t>
 </t>

> Trond comments:
> The whole section needs justification: there is no discussion anywhere
> of why a server implementor might prefer to receive COMMIT requests
through
> the MDS. 

I hope I've done that.

> Why couldn't such a server simply perform an implicit COMMIT
> on LAYOUTCOMMIT since the latter has persistence guarantees anyway on
> the metadata?

What happens if the data server goes down between the write and the 
LAYOUTCOMMIT?  COMMIT has the apparatus to deal with that.  That's
what it was designed for.

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 08 Nov 2007 15:45:39 -0500

> > of why a server implementor might prefer to receive COMMIT requests

Yup. Looks a lot better.

> what it was designed for.

If the data server goes down, then the MDS can (and probably should)
recall the layout for that fsid. This will normally trigger the same
recovery procedure as a COMMIT would.

In fact, for the case where the COMMIT would be sent to the MDS, the
client can probably optimise better if the server issues the layout
recall by fsid, since the fsid will identify exactly which server went
down, and therefore which stripes need rewriting.
With COMMIT, you have no info about which DS rebooted, and so the client
needs to write all stripes.

Trond

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 8 Nov 2007 16:28:05 -0500

> recovery procedure as a COMMIT would.

My head is hurting.

So you do the LAYOUTCOMMIT, and if you had not done a COMMIT to the DS's
before (in which case this would work but your performance would suck),
the MDS would note the DS went down and return DELAY.  Then it would do
a layout recall and then you would do the COMMITs to the various DS's
and would followup with rewrites on those you needed to.

Now you are going to do the LAYOUTRETURNs but you have to the
LAYOUTCOMMIT
before you do the LAYOUTRETURN.  Now what is making my head hurt is
dealing
with all the possiblities of various failures at this point.  What
happens
if another DS fails at this point.  You can't recall the LAYOUT any
more.
It is already recalled.

Maybe there is a way around this, and the issue of preventing the
DSs from being too agressive in doing COMMITs, i.e. doing them at all
when there is nothing specific prompting them.  But I don't see us
tying this up and putting a pretty bow on it in time for a draft for
the next IETF meeting.

It seems to me that this option is reasonable and can be reasonably
explained in the spec, and while I won't say that you cannot figure
out how to avoid it, I don't know that you can do that either.

My preferance would be to explain it as suggested and give the people
who believe that it can be dispensed with and should be dispensed with 
the option for arguing for that as part of last call.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 08, 2007 3:46 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Proposal for <cref> relative to MDS COMMIT

Yup. Looks a lot better.

Trond

---

From: Trond Myklebust <[email-masked]>
Date: Thu, 08 Nov 2007 16:56:32 -0500

> and would followup with rewrites on those you needed to.

Bear in mind that we are only talking about the case of COMMIT-to-MDS.

However, I see that I was confused. I didn't really mean a recall of the
fsid, but of the deviceID. Are we, or are we not going to support
recalling deviceIDs?
There was some discussion about doing this, since Marc in particular
wanted the ability to invalidate a deviceID without recalling the layout
(the client would presumably re-issue a GETDEVICEINFO request and get
directed to a new DS) but I've lost track of the status of that request.

Trond

---

From: Robert Gordon <[email-masked]>
Date: Thu, 08 Nov 2007 16:06:06 -0600

> the option for arguing for that as part of last call.

I prefer your suggested text.

Robert.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 08 Nov 2007 19:09:13 -0600

> directed to a new DS) but I've lost track of the status of that  

Deviceid will have a stateid that is recallable and we are closing
on having deviceid split with major/minor such that the client will
know the groupings of deviceids under a single stateid.

See the recent threads on this topic for more detail.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 9 Nov 2007 10:55:48 -0500

> the option for arguing for that as part of last call. 

In the absence of violent objections, I'm going ahead and putting 
the text discussed into the spec.

> However, I see that I was confused. I didn't really mean a recall of 
> the fsid, but of the deviceID. Are we, or are we not going to support 
> recalling deviceIDs?

I can't predict the future.  The only thing I can do (barely) is to 
read the current version of the spec based on the CVS repository and
in there I don't see any provision for recalling a specific device
(and somehow implicitly recalling all layout segment for that device
which it seems like you are depending on).

> directed to a new DS) but I've lost track of the status of that
request.

I think that that is the notify.

Device maps can also be recalled but there you are recalling all device
with a given major id which may be too coarse-grained for what you want
(or you may be able to arrange things so it isn't).  The effect on 
existing layouts (whether there is an implicit recall) is not clear at
all.  One problem is that although by giving device maps stateids, we
have implicitly provided a recall mechanism, the text for CB_RECALL 
only mentions recall of delegations, and so the status and semantics
of recalls of device maps isn't very clear.

I'm also having trouble with this text:

   Device ID mappings represent another form of stateid
   Section 8.2.1.  The GETDEVICEINFO and GETDEVICELIST operations each
   return a device stateid. 

My assumption is one stateid per major device. 

   Like file delegations, the device stateid
   is recallable.  A recall of the device stateid will remove or
   invalidate the device ID mappings 

are "remove" and "invalidate" intended to be synonyms or are these two
different things which can be done?

   as well as lease expiration.  

Huh?

   The
   GETDEVICEINFO and GETDEVICELIST operations update the current
   filehandle to facilitate the recall of the device stateid. 

I've already asked about this with regard to text for the ops
GETDEVICEINFO and GETDEVICELIST.  I still have no clue what is
intended.

I'm not sure but if look to me like to do what you want the the MDS 
would have to do layout recalls on each of the individual stripes 
that the device covered.

-----Original Message-----
From: Trond Myklebust [mailto:[email-masked]] 
Sent: Thursday, November 08, 2007 4:57 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: RE: [nfsv4] Proposal for <cref> relative to MDS COMMIT

On Thu, 2007-11-08 at 16:28 -0500, Noveck, Dave wrote:
> > > What happens if the data server goes down between the write and 
> > > the LAYOUTCOMMIT?  COMMIT has the apparatus to deal with that.  
> > > That's what it was designed for.
> 

> So you do the LAYOUTCOMMIT, and if you had not done a COMMIT to the 
> DS's before (in which case this would work but your performance would 
> suck), the MDS would note the DS went down and return DELAY.  Then it 
> would do a layout recall and then you would do the COMMITs to the 
> various DS's and would followup with rewrites on those you needed to.

Trond

> LAYOUTCOMMIT before you do the LAYOUTRETURN.  Now what is making my 
> head hurt is dealing with all the possiblities of various failures at 
> this point.  What happens if another DS fails at this point.  You 
> can't recall the LAYOUT any more.
> It is already recalled.
> 
> Maybe there is a way around this, and the issue of preventing the DSs 
> from being too agressive in doing COMMITs, i.e. doing them at all when

> there is nothing specific prompting them.  But I don't see us tying 
> this up and putting a pretty bow on it in time for a draft for the 
> next IETF meeting.
> 

> > > Why couldn't such a server simply perform an implicit COMMIT on 
> > > LAYOUTCOMMIT since the latter has persistence guarantees anyway
> on
> > > the metadata?
> > 

> With COMMIT, you have no info about which DS rebooted, and so the 
> client needs to write all stripes.
> 
> Trond

---

From: Mike Eisler <[email-masked]>
Date: Sat, 10 Nov 2007 14:17:53 -0800 (PST)

I am unnerved by the seemingly unstoppable drive to having
zillions of stateids for device ID to device address
mappings, and still don't understand why we are going in
that direction. The latest I've read is that there will
be only one stateid per major device ID. Given that a major
device ID has ballooned to 64 bits, that does not comfort
me. The current path means that device ID notifications
have little, if any, utility, given that delegations for
2^64 different major device IDs can be handed out.

What are the requirements? No one has put them down in a
concise list list as far as I know. So I will state them,
based on what I've read and heard since the we had a pNFS
inspection meeting during the Austin bake-a-thon.

Requirements:

- Server

-- Without forcing the client to re-issue GETDEVICELIST,

--- be able to invalidate a single device ID's
    address mappings

--- Be able to change a single device ID's
    address mappings

--- Be able to change add a single device ID's
    address mappings

-- Allow recall of layouts by FSID, and possibly other
   coarse grained objects. My understanding is that for
   some layout types and/or pNFS server implementations
   this is critical.

- Client

-- Do not burden the general pNFS implementation
   with aspects that specific to a pNFS server or
   implementation (e.g. the possibility of recall by
   FSID requires clients to organize layouts by FSID,
   which is really a kick in the teeth if the server
   never issues such a recall.

-- If there must be a recall by a coarse grained object,
   don't force the client implementation to understand what
   that object is. Every time we add a layout type or
   a pNFS server implementation should not be an occasion
   to re-write client's generic pNFS layer.

- Both client and server

-- Minimize the cost implementing these requirements.
   E.g. do not go down the road of requiring multiple
   stateids for devices.

So here is the outline of what I've proposed before,
in more detail this time:

- DeviceIDs have a major and minor component. The current
  consensus that they be the same size and format as the
  fsid makes sense, since it makes it easier for servers
  to implement recall by fsid.

- Replace layout recall by FSID with layout recall by major
  device ID. Now the server can organize its data how it
  sees fit without giving the client information it doesn't
  need.

- Allow servers to indicate that they don't require
  clients to organize layouts by major device ID by
  encoding the major device ID such that if

  	did_major & 0x8000000000000000 is FALSE

  then this is an indication the device ID is to be
  treated as a flat object and no recall by major device
  ID will occur.

- There is no more than one device stateid per
  clientID/layout type pair. It is returned by
  GETDEVICEINFO, and GETDEVICELIST.

- enhance GETDEVICELIST to optionally take a major device
  ID. This will allow the client to obtain all the minor
  device IDs for a major device ID as well as the address
  mappings. This is actually the only new thing I've added.

- The preferred way device ID mappings are updated is through
  notification by CB_NOTIFY (or the server can recall
  the device stateid, and force the client to re-issue 
  GETDEVICELIST/GETDEVICEINFO for all mappings). After
  CB_NOTIFY is received, the client can issue GETDEVICELIST
  or GETDEVICEINFO as apporpriate.
  CB_NOTIFY has x deviceID notifications:

-- NOTIFY4_DEVICE_NEW_MAJOR: a new major device ID has been
   added. If the client cares, it can issue GETDEVICELIST
   to obtain it.

-- NOTIFY4_DEVICE_ADDED_TO_MAJOR: one or more minor
   devices added to the major device. If the client cares,
   it can issue GETDEVICELIST to get the current minor
   devices for the major device.

-- NOTIFY4_DEVICE_MAJOR_DELETED: some or all device IDs
   with the specified major device ID are deleted. If
   just some device IDs are deleted, the client can use
   GETDEVICELIST with the major device ID to see what
   is left.

-- NOTIFY4_DEVICE_ID_ADD - full deviceID, as in draft-15
-- NOTIFY4_DEVICE_ID_CHANGE - full deviceID, as in draft-15
-- NOTIFY4_DEVICE_ID_DELETE - full deviceID, as in draft-15
  
  These notifications are asynchronous. It is up to the
  pNFS server to prevent the use of deleted or updated
  device IDs.  We have at our disposal:

  - fencing (preferred)
  - the SLA method that the blocks layout type
    OPTIONALly specifies (if necessary)

  For those who argue that recall by a per major-device
  ID is safer, I see the same race conditions, and the
  same solutions: fencing and SLA.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 11 Nov 2007 09:56:26 -0500

> zillions of stateids for device ID to device address mappings, 
> and still don't understand why we are going in that direction.

"zillions" is imprescise. 

The current proposal is to allow 256 undecillion device ids
but note that a given client ID can only have 64 octillion
statids.

If someone proposes expanding the stateid, I will try to stay
calm.  I don't promise to succeed however. 

> The latest I've read is that there will be only one stateid 
> per major device ID.

That's been my assumption but nobody has either confirmed or
contradicted that understanding.  Some people may be thinking
about a stateid for a single device id.

>    never issues such a recall.

The client has to organize his data structure by fsid's for
many reasons.  He can get a MOVED error, for instance.  He
has a choice of linking together his file objects or searching
by fsid when the event happens but he has to have enough
information to do that in any case.   

>   sees fit without giving the client information it doesn't

But at a large cost in client complexity.  The difference here
is that a recall by FSID is the union of a set of recalls by
file, i.e. you are recalling the entire layout for some set of 
files whereas with a recall by device id or major device id, you 
may be recalling pieces of the layout for one file and different
pieces for a different file, etc.

Then you have to return them.  With a file layout, we have a  
single compact striping pattern that implies the layout for 
millions of distinct byte ranges.  If you have a way to abbreviate 
that, you still have the issue of imposing on the client a very 
complicated state space for what layouts it has.  For files, what 
you would like is either "I have the whole layout" or "I don't 
have any of the layout" and without this kind of stuff you might 
be able to get away with it.  Also, the logic in the server to 
determine when the client has returned all of the requested layout 
ranges doesn't seem very nice.

>   treated as a flat object and no recall by major device

Why do we need this?  If I want to not have the client deal 
with multiple major device id's, why don't I just use  
only one (major id equals zero).  The server still has 16 
quintilion devices to work with.  Why isn't that enough?

>   GETDEVICEINFO, and GETDEVICELIST.

Works for me but for others the issue is going to be 
the effect/validity of recall on this stateid.  And if
it is not usefully recallable, what is the point of 
giving this a stateid?

We also need to resolve the issue cfh with regard to 
GETDEVICEINFO and GETDEVICELIST.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Saturday, November 10, 2007 5:18 PM
To: [email-masked]
Subject: [nfsv4] device IDs and stateids

I am unnerved by the seemingly unstoppable drive to having zillions of
stateids for device ID to device address mappings, and still don't
understand why we are going in that direction. The latest I've read is
that there will be only one stateid per major device ID. Given that a
major device ID has ballooned to 64 bits, that does not comfort me. The
current path means that device ID notifications have little, if any,
utility, given that delegations for
2^64 different major device IDs can be handed out.

What are the requirements? No one has put them down in a concise list
list as far as I know. So I will state them, based on what I've read and
heard since the we had a pNFS inspection meeting during the Austin
bake-a-thon.

Requirements:

- Server

- Client

- Both client and server

So here is the outline of what I've proposed before, in more detail this
time:

  

  - fencing (preferred)

--- "Noveck, Dave" <[email-masked]> wrote:

> Device maps can also be recalled but there you are recalling all 
> device with a given major id which may be too coarse-grained for what 
> you want (or you may be able to arrange things so it isn't).  The 
> effect on existing layouts (whether there is an implicit recall) is 
> not clear at all.  One problem is that although by giving device maps 
> stateids, we have implicitly provided a recall mechanism, the text for

> CB_RECALL only mentions recall of delegations, and so the status and 
> semantics of recalls of device maps isn't very clear.
> 

> would have to do layout recalls on each of the individual stripes that

> the device covered.

---

From: Spencer Shepler <[email-masked]>
Date: Sun, 11 Nov 2007 11:41:35 -0600

> about a stateid for a single device id.

Stateid per deviceid's major.

> pieces for a different file, etc.

It seems there is value in allowing for LAYOUTRECALL by fsid
in that the layouts may be changing but the deviceid mappings
may stay the same.  This allows for flexibility (some may argue
to much flexbility) and should allow for future server implementations
as well as future layout types that may need something like this.

Given that the deviceid's stateids have been described as
delegations on the mapping, the use of CB_RECALL would be used
for recalling the deviceid mappings associated with the stateid
and the client would signify their return with DELEGRETURN.

> quintilion devices to work with.  Why isn't that enough?

I agree that we should not special case this.  The client
will need to deal, in the general case, of a server that
has multiple deviceid.major values and a server that has
but a single deviceid.major will be the degenerate case and
likely not going to lead to significant optimizations
by the client.

Spencer

> GETDEVICEINFO and GETDEVICELIST.

The issue being that the current filehandle is captured
by the client to deal with CB_RECALL and DELEGRETURN
processing?  To simplify the GETDEVICELIST results,
I would suggest that there is a single "deviceid mapping"
filehandle that is used by the server that is set
as current filehandle for the results.

Spencer

>> that

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 11 Nov 2007 14:24:20 -0500

> delegations on the mapping, the use of CB_RECALL would be 
> used for recalling the deviceid mappings associated with the 
> stateid and the client would signify their return with 
> DELEGRETURN.

That's reasonable but I think we need some text for CB_RECALL
to make that explicit.  I think some of the confusion about
cfh on GETDEVICEINFO and GETDEVICELIST  may be related to
the fact that device stateis have been described as delegations
on the mapping without being terribly clear on details of 
that analogy/identification.

To clarify, what is the effect of DELEGRETURN on layouts that
contain the references to recalled device mappings?  Am I 
correct that there is no recall (i.e. requirement to actually
do a LAYOUTRETURN), but ...
   
    the client is obliged not to use the old mappings

    the server is obliged not to allow the old mappings to be
    used.

    When the clients new mappings for the same device id
    it may use existing layouts for those device ids.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Sunday, November 11, 2007 12:42 PM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] device IDs and stateids

On Nov 11, 2007, at 8:56 AM, Noveck, Dave wrote:

>
>> The latest I've read is that there will be only one stateid per major

>> device ID.
> contradicted that understanding.  Some people may be thinking about a 
> stateid for a single device id.

> The client has to organize his data structure by fsid's for many 
> reasons.  He can get a MOVED error, for instance.  He has a choice of 
> linking together his file objects or searching by fsid when the event 
> happens but he has to have enough information to do that in any case.
>
> But at a large cost in client complexity.  The difference here is that

> a recall by FSID is the union of a set of recalls by file, i.e. you 
> are recalling the entire layout for some set of files whereas with a 
> recall by device id or major device id, you may be recalling pieces of

> the layout for one file and different pieces for a different file, 
> etc.

It seems there is value in allowing for LAYOUTRECALL by fsid in that the
layouts may be changing but the deviceid mappings may stay the same.
This allows for flexibility (some may argue to much flexbility) and
should allow for future server implementations as well as future layout
types that may need something like this.

Given that the deviceid's stateids have been described as delegations on
the mapping, the use of CB_RECALL would be used for recalling the
deviceid mappings associated with the stateid and the client would
signify their return with DELEGRETURN.

> Why do we need this?  If I want to not have the client deal with 
> multiple major device id's, why don't I just use only one (major id 
> equals zero).  The server still has 16 quintilion devices to work 
> with.  Why isn't that enough?

I agree that we should not special case this.  The client will need to
deal, in the general case, of a server that has multiple deviceid.major
values and a server that has but a single deviceid.major will be the
degenerate case and likely not going to lead to significant
optimizations by the client.

Spencer

> Works for me but for others the issue is going to be the 
> effect/validity of recall on this stateid.  And if it is not usefully 
> recallable, what is the point of giving this a stateid?
>
> We also need to resolve the issue cfh with regard to GETDEVICEINFO and

> GETDEVICELIST.

The issue being that the current filehandle is captured by the client to
deal with CB_RECALL and DELEGRETURN processing?  To simplify the
GETDEVICELIST results, I would suggest that there is a single "deviceid
mapping"
filehandle that is used by the server that is set as current filehandle
for the results.

Spencer

> list as far as I know. So I will state them, based on what I've read 
> and heard since the we had a pNFS inspection meeting during the Austin

---

From: Mike Eisler <[email-masked]>
Date: Sun, 11 Nov 2007 12:01:37 -0800 (PST)

--- "Noveck, Dave" <[email-masked]> wrote:

>  I think some of the confusion about

Before I go catch a bus, the cfh for GETDEVICE* is likely an
artificial construct by the server. No cfh is required on input to
to GETDEVICE*, but a cfh is generated. This cfh is used when
the client returns the device stateid (or if one of zillions of device
stateids if trends continue), and also on recall of the device
stateid(s).

We have a delegation and recall mechanism; we shouldn't implement
a parallel system that does the same thing.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 11 Nov 2007 15:17:13 -0500

Right now, the text/synopsis for these is written as if they
used a cfh.  The error table doesn't have NOFILEHANDLE for
these but it does have FHEXPIRED.

I agree we should have the same mechanism but what these
sorts of recalls do is different enough that the text needs
make explicit the exact semantics of the two sorts of recalls
you can do through CB_RECALL. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Sunday, November 11, 2007 3:02 PM
To: Noveck, Dave; Spencer Shepler
Cc: [email-masked]
Subject: RE: [nfsv4] device IDs and stateids

--- "Noveck, Dave" <[email-masked]> wrote:

> cfh on GETDEVICEINFO and GETDEVICELIST  may be related to the fact 
> that device stateis have been described as delegations on the mapping 
> without being terribly clear on details of that 
> analogy/identification.

Before I go catch a bus, the cfh for GETDEVICE* is likely an artificial
construct by the server. No cfh is required on input to to GETDEVICE*,
but a cfh is generated. This cfh is used when the client returns the
device stateid (or if one of zillions of device stateids if trends
continue), and also on recall of the device stateid(s).

We have a delegation and recall mechanism; we shouldn't implement a
parallel system that does the same thing.

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 12 Nov 2007 00:40:39 -0600

> you can do through CB_RECALL.

Yes.  Additional text is needed.  I will work on text updates
to reflect the usage.

It would be helpful to hear from those that originally
raised these issues to ensure we are headed in the right
direction.

Spencer

> artificial

---

From: Benny Halevy <[email-masked]>
Date: Mon, 12 Nov 2007 16:51:45 +0200

> raised these issues to ensure we are headed in the right

I think you are.

Draft-16 isn't clear about the association of the device
stateid with did_major and with the filehandle "returned"
by GETDEVICE{INFO,LIST}.