Subject: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

From: [email-masked]
Date: Fri, 11 Jan 2008 14:45:46 -0500

The argument list "struct GETDEVICELIST4args" changed in draft 18
removing
        "/*  CURRENT_FH: file */".

This change made it apparent to me that in draft 15, the definition of
GETDEVICELIST was changed to "enumerate all of the device ID to device
address mappings in use by the server".   In contrast, in draft 14,
GETDEVICELIST was defined to return "all the devices associated with a
file system".

I apologize for not noticing this earlier, but the change is
significant.  IMHO, GETDEVICELIST is much more useful if the mappings it
provides relate to a given file system, rather than to all devices used
by the server.  The reason is that the former allows a client to
evaluate if it can access the devices required to do pNFS for a given
file system, whereas the later provides no such ability.  In the later
definition there is no way (short of issuing LAYOUTGET for a file) to
associate device ids with a given file system.  This is particularly
useful when a user mounts a file system.

I propose that the CURRENT_FH be added back to the definition of
GETDEVICELIST4args and that the first paragraph description be revised
to read:

   This operation is used by the client to enumerate all of the device
   ID to device address mappings in use by a file system.  An example of
   the use of this operation is for device types that use SAN devices
   and in these environments it may be helpful for a client to determine
   device accessibility upon first file system access.

   If the current file handle is the root file handle, the server should
   respond with all devices in use by the server.  On success, the
current
   filehandle retains its value.

Among other reasons, I think that the current file handle was removed
due to confusion generated by the addition in draft 15 and subsequent
removal in 18 of device stateids.

(See the thread 
http://www1.ietf.org/mail-archive/web/nfsv4/current/msg04899.html
and
http://www1.ietf.org/mail-archive/web/nfsv4/current/msg04903.html 
)

Comments?

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Fri, 11 Jan 2008 12:09:19 -0800 (PST)

I don't care either way (i.e. leave it as is, or change it). Other
opinions?

---

From: "William A. (Andy) Adamson" <[email-masked]>
Date: Fri, 11 Jan 2008 15:10:48 -0500

I vote for using the current filehandle - per fsid getdevicelist is much
more useful.

-->Andy

---

From: Marc Eshel <[email-masked]>
Date: Fri, 11 Jan 2008 13:07:18 -0800

I also prefer to keep the FH.
Marc.

"William A. (Andy) Adamson" <[email-masked]> 
01/11/2008 12:10 PM

To
[email-masked]
cc
[email-masked]
Subject
Re: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

-->Andy

On Jan 11, 2008 3:09 PM, Mike Eisler <[email-masked] > wrote:
I don't care either way (i.e. leave it as is, or change it). Other
opinions?

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 11 Jan 2008 16:58:03 -0500

Somewhere in there, it was proposed that device id's were to be
qualified by the file system, and after going round and round on that
issue we that device mappings were per server and not per filesystem and
I think we don't want to change that.

> IMHO, GETDEVICELIST is much more useful if the mappings it
> provides relate to a given file system, rather than to all 
> devices used by the server. 

I don't think we should approach this with a which-is-better approach?

We have two questions:

a) Is it useful to provide the entire list of devices for 
   a server?

   I would say yes, but I would defer to those writing clients.
   Are client implementers willing to say they don't need this?

b) Is it useful to provide the a list of the set of devices used
   for a given filesystem.

   I don't know but if clients think they need this then this 
   should be provided but there is no quarantee that every device 
   in the list will be used so that it would be legal to report 
   the same list of all devices for all fs's.

If we want both we can accommodate with a boolean such as we do with
RECLAIM_COMPLETE.

If you have lists of devices by fs, then each device may be used with a
set of fs's and then we have the issue of defining the appropriate
notifications when the set changes.  If the set of devices is fixed,
then it is easy to make the list per-fs, but when you factor in
notifications, then things start to get more difficult.  The client
winds up with as many notification streams as DETDEVICELSTs for
different fsid's and for some servers (that use a more global allocation
approach) you will wind up getting lots of duplicate device addition
messages.  You would almost certainly need some way to eliminate the
notification stream when you ceased to be interested in a given fsid.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Friday, January 11, 2008 2:46 PM
To: [email-masked]
Subject: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

(See the thread 

Comments?

---

From: [email-masked]
Date: Fri, 11 Jan 2008 17:25:17 -0500

I side with Andy and Jason.
-mw

________________________________

From: William A. (Andy) Adamson [mailto:[email-masked]] 
Sent: Friday, January 11, 2008 3:11 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

-->Andy

	I don't care either way (i.e. leave it as is, or change it).
Other
	opinions?

	--- [email-masked] wrote:
	
	> This change made it apparent to me that in draft 15, the
definition
	> of
	> GETDEVICELIST was changed to "enumerate all of the device ID
to 
	> device
	> GETDEVICELIST was defined to return "all the devices
associated with
	> a
	> significant.  IMHO, GETDEVICELIST is much more useful if the
mappings
	> it
devices 
	> used
	> file system, whereas the later provides no such ability.  In
the 
	> later
	>    This operation is used by the client to enumerate all of
the 
	> device
	>    ID to device address mappings in use by a file system.  An
example
	> of
	>    and in these environments it may be helpful for a client to

	> determine
	>    If the current file handle is the root file handle, the
server
	> should

---

From: Mike Eisler <[email-masked]>
Date: Fri, 11 Jan 2008 14:36:25 -0800 (PST)

> I think we don't want to change that.

Agreed, and at 128 bits long we don't need to.

> If we want both we can accommodate with a boolean such as we do with

I'm loathe to make a .x change if it can be avoided.
Jason has proposed the server distinguish the cases via PUTROOTFH 
(entire list) or a non-PUTROOTFH filehandle (e.g. from PUTFH).

> notification stream when you ceased to be interested in a given fsid.

I think of the fsid parameter to GETDEVICELIST as a filter,
and so with that model, I don't see why there is a notification issue.
Last notification registration wins. No duplicate notifications.

I do see an issue if a client issues GETDEVICELIST to fsid 1, with
notification mask X,  gets back device A, and issues 
GETDEVICELIST to fsid 2, with notification mask Y (where X != Y). 
The rule should be, last processed mask wins, but if the client 
isn't serializing GETDEVICELIST calls it cannot know which won. Short
of 
adding a stateid to GETDEVICELIST arguments and results (hey, I'm
joking), I think we can resolve such a race with text:

   If two unserialized GETDEVICELISTs for two 
   different fsids return the same deviceID, but different
   gdlr_notification values, the client will need to issue 
   GETDEVICEINFO on the deviceID to be sure what notifications 
   will be sent for the deviceID.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 11 Jan 2008 18:06:50 -0500

> Jason has proposed the server distinguish the cases via 
> PUTROOTFH (entire list) or a non-PUTROOTFH filehandle (e.g. 
> from PUTFH).

I'm OK with this if the distinction is based on the filehandle, i.e. if
you do a PUTFH with the root filehandle you get the global list.  If you
had to remember the op which established the current filehandle
separately for the filehandle itself,  it would be a  big pain in the
ass.  If I do PUTROOTFH/GETDEVICELIST,
PUTROOTFH/SAVEFH/PUTFH/READ/RESTOREFH/GETDEVICELIST,
PUTROOTFH/LOOKUP/LOOKUPP/GETDEVICELST, or a PUTFH with the handle I got
from PUTROOTFH/GETFH followed by a GETDEVICELIST, I should get the
global list.

> I think of the fsid parameter to GETDEVICELIST as a filter, and 
> so with that model, I don't see why there is a notification issue.

That's clear and implementable but unless I'm missing something, I don't
see how that is adequate.  If a client can be interested in two
filesystems at the same time and if the list of associated devices for
those filesystems can both change, how is the client to know, at any
given time what those lists are, if he can only get notification about
devices for one filesystem at a time.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Friday, January 11, 2008 5:36 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

--- "Noveck, Dave" <[email-masked]> wrote:

> issue we that device mappings were per server and not per filesystem 
> and I think we don't want to change that.

> 
> If you have lists of devices by fs, then each device may be used with 
> a set of fs's and then we have the issue of defining the appropriate 

I think of the fsid parameter to GETDEVICELIST as a filter, and so with
that model, I don't see why there is a notification issue.
Last notification registration wins. No duplicate notifications.

I do see an issue if a client issues GETDEVICELIST to fsid 1, with
notification mask X,  gets back device A, and issues GETDEVICELIST to
fsid 2, with notification mask Y (where X != Y). 
The rule should be, last processed mask wins, but if the client isn't
serializing GETDEVICELIST calls it cannot know which won. Short of
adding a stateid to GETDEVICELIST arguments and results (hey, I'm
joking), I think we can resolve such a race with text:

> significant.  IMHO, GETDEVICELIST is much more useful if the mappings 
> it provides relate to a given file system, rather than to all devices 
> used by the server.  The reason is that the former allows a client to

---

From: Mike Eisler <[email-masked]>
Date: Fri, 11 Jan 2008 15:30:50 -0800 (PST)

> devices for one filesystem at a time.

Ok, now I am understanding Dave's issue.

I think the client should be able to get notifications for
more than one filesystem at a time. E.g.:

  PUTFH fh1, GETDEVICELIST, PUTFH fh2, GETDEVICELIST

can establish notifications for all the deviceIDs returned
by each GETDEVICELIST.

That works fine for DELETE and CHANGE notifications.

The problem is what does it mean for ADD?

ADD takes no fsid as an argument. We don't want to add one
because deviceIDs can be shared across fsids.

So how does a client that is using GETDEVICELIST-by-fsid determine
what fsids a deviceID sent in a notification works with?

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 11 Jan 2008 21:48:57 -0500

> can establish notifications for all the deviceIDs returned

But if the client ceases to be interested in deviceID notifications for
the filesystem for fh2 (unmount), how is this communicated to the
server?

> what fsids a deviceID sent in a notification works with?

Yes, but I'm worried it may be worse than that.  If deviceIDs can be
shared across fsids, do we have any basis to say that the set of fsids
with which a device can be associated with is fixed, even in the absence
of the addition of a deviceID.  If at time T, deviceID A is associated
with fsid X and not Y, can we assume that it will never be associated
with fsid Y, and if we cannot so assume, how do we communicate to the
client that that change has occurred?

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Friday, January 11, 2008 6:31 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

--- "Noveck, Dave" <[email-masked]> wrote:

> > I think of the fsid parameter to GETDEVICELIST as a filter, and so 
> > with that model, I don't see why there is a notification issue.
> That's clear and implementable but unless I'm missing something, I 
> don't see how that is adequate.  If a client can be interested in two

---

From: Mike Eisler <[email-masked]>
Date: Sat, 12 Jan 2008 06:31:50 -0800 (PST)

OK.

So to those proponents of per-fsid GETDEVICELIST, we have a problem
if we are going to support that concept.

Dave has made a convincing argument that a per-fsid GETDEVICELIST
is only meaningful if the notifications are per-fsid.

If so, that leads us to the following questions/issues:

- how does a client cancel the notifications for an fsid?
- when a client gets an ADD notification, the client does not
  know what fsid the ADD is for.
- if a client has done PUTFH, GETDEVICELIST, and then does
  PUTROOTFH, GETDEVICELIST, each with notifications enabled, does
  this mean the per-fsid notifications are turned off?

(others?)

If we are to keep the existing XDR definitions, and support
per-fsid GETDEVICELIST, then I think
that leaves us with the following conclusion:

   Once a client issues a GETDEVICELIST for an fsid, the
   server MUST disable all notifications.

--- "Noveck, Dave" <[email-masked]> wrote:

> so 
> > with
> isn't
> > > used by the server.  The reason is that the former allows a
> client
> > to 
> > > definition there is no way (short of issuing LAYOUTGET for a
> file)
> > to 
> > >    If the current file handle is the root file handle, the server
> 
> > > should

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 12 Jan 2008 12:29:10 -0500

BTW, the current definition of devlist_item4 in section 3.3.16 says

> An array of these values is returned by the GETDEVICELIST operation.
> They define the set of devices associated with a file system for the
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> layout type specified in the GETDEVICELIST4args. 

And section 18.40.3 says:
> The CB_NOTIFY_DEVICEID operation is used by the server to send
> notifications to clients about changes to pNFS device IDs.  The
> registration of device ID notifications occurs when the device
> mapping stateid is established using GETDEVICEINFO or GETDEVICELIST.

> is only meaningful if the notifications are per-fsid.

I'm not sure I'm arguing that, but I am arguing that if we have per-fsid
GETDEVICELIST, we have to address the issue of notifications somehow.
What, if anything, is required as far as per-fsid notifications depend
on what we are trying to achieve with per-fsid GETDEVICELIST.

So here is my proposal for discussion:

     Current fh is part of the input for GETDEVICELIST.

     If current fh is the server root, request is for global list.

     If the current fh is anything else, request is for the 
     filesystem associated with current fh.

     For handling ADDITION notifications, the server defines for every
     client a set of interest and only reports additions to that set.
     It does not report additions to specific subsets of that set of
     interest.

     If ADDITION has been specified by GETDEVICELIST for the global
     list, then ADDITION notifications only reflect additions to the
     global list and there are no notifications regarding changes in
     use by specific filesystems.

     If ADDITION has not been specified by GETDEVICELIST for the global
     list, the set of interest consists of the set of devices for all
the
     fs's for which ADDITION notification has been specified.  When a 
     device enters this set, an ADDITION notification occurs but there
     is no notification when there is change in the set of fs's that a
     given device is used with, if it doesn't cause a device to enter
     or to leave that set.

     Similar principles apply to DELETE and CHANGE notifications, but
     we have the additional issue of notifications based on
GETDEVICEINFO.
     To specify this, consider separately the set of interest discussed
     above, which we will call the set of GDL-interest and the set of
     devices for which notification of the appropriate type was
specified
     via GETDEVICEINFO, which we will call the set of GDI-interest.

     A CHANGE notification occurs when there is a change to a device 
     which belongs either to the set of GDI-interest or GDL-interest.
     The notification contains no indication as to which of these (it
     could be both) applies.

     In managing the sets GDI-interest and GDL-interest a server, may
     in the interest of limiting storage use, not keep precise note of 
     the exact lists, so long as any device the client has requested
     information about, is noted.  In an extreme case, it may decide
     to make these sets equal to the set of all devices that the server
     has access to.  The client should ignore ay notification about
     unknown devices which can result from this policy.  

     A deletion notification occurs when a device which was in the set
     of GDL-interest leaves that set, and the device is not in the set
     of GDI-interest, or, when a device which is in the set of interest
     ceases to exist.  The notification contains no indication of which
     of these is the case.

     A GETDEVICEINFO or GETDEVICELIST can update the notification mask
     associated with a specific device, with the device set of an fs,
     or with the global list.  In particular, if the set of
notifications
     requested is empty, any previous request for notifications for that
     entity is cancelled.

     If maxcount for GETDEVICELIST is specified as zero, then no device 
     data is returned, and notifications are appropriately updated, but
     NFS4ERR_TOOSMALL is not returned.  This allows easy cancellation of
     notifications, particularly per-fs notifications.  Note that if 
     maxcount is too small but isn't zero (e.g. it is one), then 
     NFS4ERR_TOOSMALL will be returned. 

     Note that removing notifications for GETDEVICEINFO will not result
     in a deletion notification, whereas when the global device list
     has no notifications specified, removal of notifications for one
     file system will cause DELETION notifications for all devices that
     were used by that file system and are not used by any other fs
     in the remaining set of GDL-interest.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Saturday, January 12, 2008 9:32 AM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

OK.

So to those proponents of per-fsid GETDEVICELIST, we have a problem if
we are going to support that concept.

Dave has made a convincing argument that a per-fsid GETDEVICELIST is
only meaningful if the notifications are per-fsid.

(others?)

If we are to keep the existing XDR definitions, and support per-fsid
GETDEVICELIST, then I think that leaves us with the following
conclusion:

--- "Noveck, Dave" <[email-masked]> wrote:

> > I think the client should be able to get notifications for more than

> > one filesystem at a time. E.g.:
> >
> > can establish notifications for all the deviceIDs returned by each 
> > GETDEVICELIST.
> 
> But if the client ceases to be interested in deviceID notifications 
> for the filesystem for fh2 (unmount), how is this communicated to the 
> server?
> 
> > ADD takes no fsid as an argument. We don't want to add one because 
> > deviceIDs can be shared across fsids.
> 

> with which a device can be associated with is fixed, even in the 
> absence of the addition of a deviceID.  If at time T, deviceID A is 
> associated with fsid X and not Y, can we assume that it will never be 
> associated with fsid Y, and if we cannot so assume, how do we 
> communicate to the client that that change has occurred?
>  
> 
> I think the client should be able to get notifications for more than 
> one filesystem at a time. E.g.:
> 
> can establish notifications for all the deviceIDs returned by each 
> GETDEVICELIST.
> 
> ADD takes no fsid as an argument. We don't want to add one because 
> deviceIDs can be shared across fsids.
>

---

From: Robert Gordon <[email-masked]>
Date: Sat, 12 Jan 2008 13:38:39 -0600

>         "/*  CURRENT_FH: file */".

We sat in the room (in Vancouver) and decided that devices would not  
be tied
to filesystems. I see no compelling reason to alter that decision. We  
need
to stop flip-flopping.

-- 
Robert...
"Procrastination is the thief of time"

---

From: Mike Eisler <[email-masked]>
Date: Sat, 12 Jan 2008 13:45:05 -0800 (PST)

> So here is my proposal for discussion:

Looks very complex, and I didn't understand it.

Here is something I think is simpler.

      Current fh is part of the input for GETDEVICELIST.
      If current fh is from PUTROOTFH, request is for global list.
      If the current fh is anything else, request is for the 
      filesystem associated with current fh. Note that since
      GETDEVICELIST may require multiple passes to return the
      global list or the per-fsid list, clients that change the
      current FH in a pass are in error and get undefined 
      results, though the server MAY return NFS4ERR_INVAL 
      if it detects this erroneous condition.

      There are three notification events, ADD, DELETE, and CHANGE.

      ADD can only be registered via GETDEVICELIST and only if
      the current FH is from PUTROOTFH. ADD applies to the global
      list.

      A client can register or unregister ADD events only
      via GETDEVICELIST (and only if the current FH is from
      PUTROOTFH). Attempts to do so via GETDEVICEINFO are
      ignored.
      
      A client can register or unregister DELETE and CHANGE events
      via GETDEVICEINFO and GETDEVICELIST.

      When DELETE or CHANGE event is registered, it applies to all
      deviceIDs. When a DELETE or CHANGE event is unregistered, it
      applies to all deviceIDs.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 13 Jan 2008 13:52:36 -0500

I wasn't in Vancouver and can't speak to what was decided there, but,
especially as this change would complicate the notification logic (how
much is still to be decided), can we have a clear explanation of why
this change is needed? 

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Saturday, January 12, 2008 2:39 PM
To: NFSv4
Subject: Re: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

>         "/*  CURRENT_FH: file */".

We sat in the room (in Vancouver) and decided that devices would not be
tied to filesystems. I see no compelling reason to alter that decision.
We need to stop flip-flopping.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 13 Jan 2008 13:52:36 -0500

Mike Eisler wrote:

> Looks very complex, 

It definitely is.  I think it does provide notifications to fully
support per-fs GDL without an XDR change, but the price is specification
complexity.  On the other hand, while there is implementation
complexity, supporting it does look doable from what I can determine.

> and I didn't understand it.

Will provide crib notes below.

> Here is something I think is simpler.

I was going in this direction when I discovered that it is at least
possible to provide per-fs notifications without an XDR change if the
notification was whether the device was used by the set of fs's the
client was interested in, as long as you don't have to tell him which
device is used by which fs's.

>     If current fh is from PUTROOTFH, request is for global list.

Would prefer "is the server root fh".

>     if it detects this erroneous condition.

This seems too harsh.  If something interrupts fully enumerating a list,
then I think it is OK to say you have given up on that list, but giving
the client an error seems to much.  Also, how do you get out of this
condition?  I assume the error doesn't change the servers last-GDL-fh
(for example, getting NFS4ERR_TOOSMALL wouldn't change this so you can
properly recover) so can you ever get out of this?

Also, in this proposal, the client may be doing a global GDL just to
register his interest in notifications. With this provision, he would
have to read the entire global list, which he isn't interested in.
Similarly for indicating a change in the set of notifications -- an
unbounded set of GDL requests for which the data may be thrown away.  

>     the current FH is from PUTROOTFH. ADD applies to the global

This means that if I get a list of devices for fs A, then there is no
way to be notified to additions to that set, if the device is not new to
the global list.  I think "Deal with it" is a reasonable answer here,
but the point is that this approach gives you per-fs GDL without
meaningful notification support for that feature.

>     PUTROOTFH). Attempts to do so via GETDEVICEINFO are

Agreed as far as GDI.  I had assumed that, but I suppose that should be
made explicit.

>     applies to all deviceIDs.

Including the case of GDI??  I had assumed that GDI notifications only
applied to the specific device gotten-info and I don't see what about
per-fs GDL would change that.  In particular, a reasonable model for the
client would be not to use GDL at all, but to use GDI for all unknown
devices it encounters and it would be nice if the client got appropriate
DELETE and CHANGE notification for the set of devices it had gotten-info
about.

There are a few issues about when the event registration happens:

     If a pass is multi-request, does the registration change 
     occur at the start or the end of the pass?

     What about the case of the registration mask being different 
     in different requests of a pass?  How is this dealt with?

To summarize, I think something along these lines is reasonable but,

     I'm not sure it meets the needs for notification that underlay
     the device management approach that would be used by clients
     using per-fs GDL.

     On the other hand, I'm not totally clear what those needs 
     actually are, or even if we need per-fs GDL at all.

So here is my alternative, admittedly more complicated approach, with

>    filesystem associated with current fh.

No big disagreement so far.

>    It does not report additions to specific subsets of that set of

So this is the essence of full notification support for per-fs GDL.  A
client is either interested in all devices or devices only for a set of
fs's, and if the latter, we can notify him when devices enter or leave
that set, as opposed to telling him when devices enter or leave the set
of devices for each of the individual fs.  That would require and XDR
change and would wind up making each fs, from the device notification
point essentially a sub-server and I don' think we want to go there. 

>    use by specific filesystems.

If you are interested in all devices, then you want to know about
additions to the set of devices.

the
>    given device is used with, if it doesn't cause a device to enter

If you are not interested in all devices but only those for a set of
fs's (i.e. you only want to keep data structures for that set of
devices) hen that is what you want to get ADDITION events for: when a
device whether it has existed previously in the global set of not, enter
the set you are interested in and what to keep data structures for. 

GETDEVICEINFO.
specified
>    via GETDEVICEINFO, which we will call the set of GDI-interest.

Assuming ADDITION does not apply to GDI notifications, just as you do.

Also assuming that we can have DELETE and CHANGE notifications for
individual devices through GDI, and that may be a point on which we
disagree.

>    The notification contains no indication as to which of these (it

For me, this is extrapolation of what exists now.  If you haven't
specified change notifications in a GDL then you will get notifications
of all of the devices for which GDI specified CHANGE notification.

>    unknown devices which can result from this policy.

This allows more complicated tracking of what clients are interested in,
without excessive worry about storage requirements.  If 100 clients each
ask for CHANGES/DELETES notifications to 10,000 devices each, then the
server should be free to turn on a bit in the structure for each client
that says send everything.  The client should not panic.  If the server
does that too much it is a quality-of-implementation issue.    

>    ceases to exist.  The notification contains no indication of which

This sounds complicated.  That's because it is complicated :-) and I
don't know how to make it less complicated :-(.

The constraints here derive from the fact that if the client is
maintaining data structures on a set of devices for fs's of interest it
should get deletion notifications when device leave that set, including
cases when this occurs because the client ceases ot be interested in a
particular fs (he has no way of doing that himself since he doesn't know
which of the devices are used by that fs and no other).  On the other
hand, we don't want a client to get a DELETE notification when indicates
that he is no longer interested in a given device by changing the mask
using GDI.

notifications
>    requested is empty, any previous request for notifications for that

If we have per-fs notifications, then we have to have a way of canceling
notifications for an fs and it seems best to generalize this so that it
applies to both GDL and GDI.

>    NFS4ERR_TOOSMALL will be returned. 

I take the approach 180 degrees away from you on this point.  Since you
will be using GDL to change/cancel notifications, where he doesn't care
about the device data at all, rather than requiring that he read it to
the end, it seems nice to allow him to specify a length of zero and not
get TOOSMALL.  He clearly know zero bytes in not enough for one entry.
He is saying he doesn't want any entries so let him do that.

>    in the remaining set of GDL-interest.

This is what the complicated specification of DELETION events above is
intended to accomplish.  I hope it does that.

---

From: Mike Eisler <[email-masked]>
Date: Sun, 13 Jan 2008 11:59:12 -0800 (PST)

> Would prefer "is the server root fh".

What if the server root is a real filesystem that is striped
over storage devices?

> This seems too harsh.  If something interrupts fully enumerating a

> list,

If you are using a cookie return from a previous pass and change
the current filehandle, I don't know what else to do.

If something interrupts enumerating a list, then the next
GETDEBVICELIST will set in the cookie argument to zero.

This is no different than READDIR. If you change the fh,
don't expect the cookies that were valid for fh1 to work for fh2.

> have to read the entire global list, which he isn't interested in.

I don't think the server needs to be implemented to see if the entire
global list is read in order to register ADD notifications. That
would be onerous. I didn't see where I wrote that.

> meaningful notification support for that feature.

Correct.

> Including the case of GDI??  I had assumed that GDI notifications

keeps it simple.

> only

I agree that GDI makes more sense, but Jason, Andy, and others see
value
in lookup by fsid in order to see if -- at the time the client 
mounts a fsid -- there is connectivity to the storage devices.
This is then used as an optimization: if there isn't connectivity,
then don't attempt LAYOUTGETs.

So given that requirement, per-fsid DELETE and CHANGE notifications
aren't needed.

>      in different requests of a pass?  How is this dealt with?

The perils of having a super-operation like GETDEVICELIST is that
these questions come up. I think the simplest is that the
last GDL or GDI wins.
 
>      actually are, or even if we need per-fs GDL at all.

Right, that's the problem, and has been an ongoing problem for
literally
years.

> point essentially a sub-server and I don' think we want to go there.

So a server arbitrarily picks a set of fsids it thinks or knows
the client is interested in, and notifications only apply to that
set?

Why is this useful? What problem is it solving? How does the server
know what fsids the client will be interested in?

> don't know how to make it less complicated :-(.

See my proposal.

Until you include an array of fsids in the ADD notification,
I don't see what problem is being solved.

And your crib notes aren't helping me.

This is starting to look like major re-design.

Jason and Andy need to weigh in.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 13 Jan 2008 23:50:20 -0500

> If something interrupts enumerating a list, then the 
> next GETDEBVICELIST will set in the cookie argument 
> to zero. 

OK.  I misunderstood you.

> What if the server root is a real filesystem that is 
> striped over storage devices?

Then use some other filehandle within that fs if you don't want the
global list.

> I don't think the server needs to be implemented to 
> see if the entire global list is read in order to 
> register ADD notifications. That would be onerous. I 
> didn't see where I wrote that.

I misunderstood.

> keeps it simple.

Yes but maybe too simple.

> > per-fs GDL would change that.  In particular, a reasonable model for

> > the client would be not to use GDL at all, but to use GDI for all 
> > unknown devices it encounters and it would be nice if the client got

> > appropriate DELETE and CHANGE notification for the set of devices it

> > had gotten-info about.
> value in lookup by fsid 

Maybe so, but so far they haven't proposed anything that would cause
problems for those not using GDL (except for those editing the spec),
but your proposal for GDI notifications would do that.

> in order to see if -- at the time the client mounts a  
> fsid -- there is connectivity to the storage devices.

People haven't been very clear about why they want this, but your
version of this doesn't make any sense to me.  You are getting a
(possibly large) list of device information and the purpose of this is
to communicate one bit of information to the client.  That doesn't seem
plausible to me.

> This is then used as an optimization: if there isn't 
> connectivity, then don't attempt LAYOUTGETs.

Is this really the intention?

> So given that requirement, per-fsid DELETE and CHANGE 
> notifications aren't needed.

I wouldn't think that per-fs GDL would be a requirement either, if this
were the goal.  Maybe a new per-fs attribute would be asked for.

> So a server arbitrarily picks a set of fsids it thinks 
> or knows the client is interested in, and notifications 
> only apply to that set?

Not arbitrarily.  The set of fs's for which notifications apply is the
set for which he has indicated he wants notification, but specifying the
notifications when doing a per-fs GDL and then not turning off
notifications for that fs via a per-fs GDL to turn off notifications for
that fs.  No ESP required. 

> Why is this useful? What problem is it solving? 

I thought the issue was that there were clients that only wanted to know
about devices for a set of mounted fs's rather than for all devices.  I
agree that we don't have much clarity on why people want per-fs GDL so
maybe I'm wrong but if that is the motivation, then they would want
corresponding notifications about change in that set of devices. 

> How does the server know what fsids the client will be 
> interested in?

He lets the client tell him.

> This is starting to look like major re-design.

Pretty much.

> Jason and Andy need to weigh in.

Definitely agree.  We need more information about why they want this and
how they would use it, in order to figure out what notification support,
if any, is required.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Sunday, January 13, 2008 2:59 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

> Would prefer "is the server root fh".

What if the server root is a real filesystem that is striped over
storage devices?

> list,
> then I think it is OK to say you have given up on that list, but 
> giving the client an error seems to much.  Also, how do you get out of

> this condition?  I assume the error doesn't change the servers 
> last-GDL-fh (for example, getting NFS4ERR_TOOSMALL wouldn't change 
> this so you can properly recover) so can you ever get out of this?

If you are using a cookie return from a previous pass and change the
current filehandle, I don't know what else to do.

If something interrupts enumerating a list, then the next GETDEBVICELIST
will set in the cookie argument to zero.

This is no different than READDIR. If you change the fh, don't expect
the cookies that were valid for fh1 to work for fh2.

I don't think the server needs to be implemented to see if the entire
global list is read in order to register ADD notifications. That would
be onerous. I didn't see where I wrote that.

> way to be notified to additions to that set, if the device is not new 
> to the global list.  I think "Deal with it" is a reasonable answer 
> here, but the point is that this approach gives you per-fs GDL without

Correct.

> Agreed as far as GDI.  I had assumed that, but I suppose that should 
> be made explicit.
>       

keeps it simple.

> per-fs GDL would change that.  In particular, a reasonable model for 
> the client would be not to use GDL at all, but to use GDI for all 
> unknown devices it encounters and it would be nice if the client got 
> appropriate DELETE and CHANGE notification for the set of devices it 
> had gotten-info about.

I agree that GDI makes more sense, but Jason, Andy, and others see value
in lookup by fsid in order to see if -- at the time the client mounts a
fsid -- there is connectivity to the storage devices.
This is then used as an optimization: if there isn't connectivity, then
don't attempt LAYOUTGETs.

>      in different requests of a pass?  How is this dealt with?

The perils of having a super-operation like GETDEVICELIST is that these
questions come up. I think the simplest is that the last GDL or GDI
wins.
 
>      actually are, or even if we need per-fs GDL at all.

Right, that's the problem, and has been an ongoing problem for literally
years.

> client is either interested in all devices or devices only for a set 
> of fs's, and if the latter, we can notify him when devices enter or 
> leave that set, as opposed to telling him when devices enter or leave 
> the set of devices for each of the individual fs.  That would require 
> and XDR change and would wind up making each fs, from the device 
> notification point essentially a sub-server and I don' think we want 
> to go there.

So a server arbitrarily picks a set of fsids it thinks or knows the
client is interested in, and notifications only apply to that set?

Why is this useful? What problem is it solving? How does the server know
what fsids the client will be interested in?

> device whether it has existed previously in the global set of not, 
> enter the set you are interested in and what to keep data structures 
> for.
> 
> specified change notifications in a GDL then you will get 
> notifications of all of the devices for which GDI specified CHANGE 
> notification.
> 
> This allows more complicated tracking of what clients are interested 
> in, without excessive worry about storage requirements.  If 100 
> clients each ask for CHANGES/DELETES notifications to 10,000 devices 
> each, then the server should be free to turn on a bit in the structure

> for each client that says send everything.  The client should not 
> panic.  If the server

See my proposal.

Until you include an array of fsids in the ADD notification, I don't see
what problem is being solved.

---

From: [email-masked]
Date: Mon, 14 Jan 2008 13:43:13 -0500

I will elaborate on why I proposed that we need per file system
GETDEVICELIST, but I'll also say up front that as Dave suggested we may
be able to handle the requirements via a new attribute.

* Being able to query the server for the list of devices required to
  use pNFS for a file system is useful on the client for the following
  reasons:

  - At mount time the user can be informed about the availablity of
    parallel access to the filesystem.  Some users will want to know
    when they mount a file system that they will be using pNFS, and
    would like to be able to diagnose connectivity errors immediately.
    User can also then specify policy, i.e. should the lack of access
    to devices be treated as a hard error in which the mount fails, or
    a soft error, in which the system will periodically try to
    establish connectivity to the required devices.

  - Diagnostic tools can be written to establish whether or not a
    given file system can be used with pNFS.

  - For block storage devices where additional SAN zoning and LUN
    masking may be required, this provides information to the
    administrator to set up the required SAN configuration.

With these requirements in mind:

* I had not considered the notification requirements (i.e. per fs)
  associated with providing a per fs GDL when I proposed the putting
  the current file handle back into the GDL call.

* I do not see a need to register for notifications on a per-file system
basis.  I think the client can do this later via GETDEVICEINFO.  Using
GETDEVICELIST with the rootfh allows a client to register or deregister
globally.  The notifications arguments on a GETDEVICELIST for a specific
filesystem would be ignored by the server.  The client can continue to
register for notifications via GETDEVICEINFO.

* A per file system attribute which returns the list of deviceids
  used by a filesystem is a workable solution.  It will force an
  additional operation (GETATTR) between the client and server,
  followed by multiple GETDEVICEINFOs, but given the availability of
  COMPOUND, it is likely that this will not require an additional
  roundtrip.

As to the question of "is this broken?" I say yes.  The original
motivation for GETDEVICELIST was to provide for the requirements above.
I agreed in Vancouver that we do not need to provide notifications per
filesystem and that device-ids were a global name space on the server,
but I missed (mea culpa) that we were removing the ability to query on a
per filesystem basics for id to address mapping information.

In summary, I am not interested in a major redesign.  The easiest and
cleanest path may be to provide an attribute which will address the
requirements listed above.  If there is agreement, I will provide
additional text describing the attribute.

-Jason

-----Original Message-----
From: Noveck, Dave [mailto:[email-masked]] 
Sent: Sunday, January 13, 2008 11:50 PM
To: [email-masked]; [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

OK.  I misunderstood you.

I misunderstood.

Pretty much.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Sunday, January 13, 2008 2:59 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

Correct.

keeps it simple.

 

 

See my proposal.

---

From: [email-masked]
Date: Fri, 18 Jan 2008 11:59:15 -0500

I spoke today with the team of linux pNFS implementers yesterday.
Besides general agreement that we want to resolve the issues associated
with GETDEVICELIST and there was also consensus that:

1.  GETDEVICE list is most useful as a per-fsid operation.  If it only
applies globally, it is not clear that it will be used.

2.  The object layout client has no intention of using the GETDEVICELIST
functionality.  The server will not implement the function, and the
client will not send the operation.

3.  The files and block clients will make use GETDEVICELIST for the
reasons mentioned below.  The reasons below are specifically about
per-file system functionality.  It is not unreasonable to imagine cases
where a client might want to know set of all devices, though I would
expect this to be the exception, rather than the norm.

4.  There was no interest in the ADD notification.  It is unclear to me
if there are any clients that intend to use this functionality.
 
5.  Clients will make use of the CHANGE notification.

6.  Clients will make use of the DELETE notifications.

7.  There was general aversion to using an attribute for the purpose of
finding the device ids used by a file system.  If we define an
attribute, we may find that there are no clients using GETDEVICELIST.

The issue of notifications, as evidenced by the email exchange, is
complex.  

I think that the best approach is that proposed by Dave in his email on
Sat 1/12/2008 at 12:29 PM.

Is there willingness to move forward with that approach?

-Jason

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, January 14, 2008 1:43 PM
To: [email-masked]; [email-masked]; [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

-Jason

-----Original Message-----
From: Noveck, Dave [mailto:[email-masked]] 
Sent: Sunday, January 13, 2008 11:50 PM
To: [email-masked]; [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

OK.  I misunderstood you.

I misunderstood.

Pretty much.

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Sunday, January 13, 2008 2:59 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

Correct.

keeps it simple.

 

 

See my proposal.

---

From: Mike Eisler <[email-masked]>
Date: Fri, 18 Jan 2008 12:11:34 -0800 (PST)

> Sat 1/12/2008 at 12:29 PM.

What Dave proposed is very complex.

If you don't need ADD, then we can take my proposal and make it
really simple.

DELETE and CHANGE are per-deviceID so nothing needs to change.

GETDEVICELIST can be made only per fsid.

ADD is removed.

DELETE ans CHANGE notifications can be set per-fsid subject to the
caveat that if a deviceID is shared by two fsids, and different
notifications are set for the deviceID, last one wins.

> would
> these
> > So this is the essence of full notification support for per-fs GDL.
> 
> > A
> know
> > >    is no notification when there is change in the set of fs's
> that
> > a

---

From: [email-masked]
Date: Fri, 18 Jan 2008 16:10:24 -0500

See below.

> Sent: Friday, January 18, 2008 3:12 PM
> To: [email-masked]

It seems like a simple solution.  With an explanation of expected client
behavior, it would work for me.

Consider the case of a client mounting multiple file systems that share
some device ids.  What should such a client do when one file system is
unmounted in order to disable future notifications?  Sending a
GETDEVICELIST message to disable the CHANGE notifications will
inadvertently cancel the notifications for the devices (shared with
another mounted file system) whois mappings are still cached.  This is
almost certainly not the desired behavior.

To avoid this, the client could:

1. Determine there are no shared deviceids, and issue a GETDEVICELIST.

2. Determine that there are shared deviceids, and use GETDEVICEINFO to
cancel the notifications for those devices that are no longer needed.

3. Make no determination at all about shared deviceids.  The client will
discard all its cached information about the deviceid to device addresss
mapping for the devices used by the unmounted file system.  It will
issue a GETDEVICELIST to disable notification for that file system.
When and if the client refetches the device id to device mappings for
one of the shared devices, it can do so with GETDEVICEINFO and it can
re-enable notifications.

In the world of the block layout, device ids are not shared between file
systems.  Object layouts will not use GETDEVICELIST.  Is this acceptable
to the implementers of the file layout?

-Jason

> > > > appropriate DELETE and CHANGE notification for the set of
devices
> > > giving the client an error seems to much.  Also, how do you get
out
> > in lookup by fsid in order to see if -- at the time the client
mounts
> > > >    fs's for which ADDITION notification has been specified.
When
> > > devices) hen that is what you want to get ADDITION events for:
when
> > > >    in the interest of limiting storage use, not keep precise
note

---

From: Marc Eshel <[email-masked]>
Date: Fri, 18 Jan 2008 13:32:11 -0800

> notifications are set for the deviceID, last one wins.

This should work for me I prefer the DELETE/CHANGE notifications to be 
per-fsid.
Marc.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 19 Jan 2008 08:10:18 -0500

> DELETE ans CHANGE notifications can be set per-fsid subject 
> to the caveat that if a deviceID is shared by two fsids, 
> and different notifications are set for the deviceID, last 
> one wins.

Note that this has the following counterintuitive/troubling consequence:

    Suppose you mount fs A and do GDL with notifications.

    Within the GDL is device X.

    Now you mount fs B and do GDL with notifications.

    Within that GDL is device X.

    Now you unmount fs B and do a GDL specifying no notifications
    because you are not interested in those devices any more.

    Notifications have been turned off for device X so if it
    is changed you don't get a notification.

Also note that if you turn on notifications for a device using GDL for a
given fs and turn them off using GDL for that same fs, you may still
have notifications set since those sets may be different and you are not
getting notifications when given devices more outside the set for a
specific fs, only when they are deleted completely or the mapping
changes.

Per-fs notifications add complexity, and you either have complexity in
the spec, or complexity for the implementation dealing with the
consequences of a simpler model, or maybe both.

I think I favor the original Eisler proposal.

My original proposal was intended to stimulate discussion and it has
done that.  The real issue is why/whether you need per-fs notifications
and so far, while I have heard explanations of why per-fs GDL is needed,
I haven't heard any of why per-fs notifications are needed.  If they are
not needed, I think we should not put them, because taking that step is
where the jump in complexity lies. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Friday, January 18, 2008 3:12 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

--- [email-masked] wrote:

> Besides general agreement that we want to resolve the issues 
> associated with GETDEVICELIST and there was also consensus that:
> 

> 2.  The object layout client has no intention of using the 
> GETDEVICELIST functionality.  The server will not implement the 
> function, and the client will not send the operation.
> 
> per-file system functionality.  It is not unreasonable to imagine 
> cases where a client might want to know set of all devices, though I 
> would expect this to be the exception, rather than the norm.
> 
> 4.  There was no interest in the ADD notification.  It is unclear to 
> me if there are any clients that intend to use this functionality.
>  
> 7.  There was general aversion to using an attribute for the purpose 
> of finding the device ids used by a file system.  If we define an 
> I think that the best approach is that proposed by Dave in his email 
> on Sat 1/12/2008 at 12:29 PM.

If you don't need ADD, then we can take my proposal and make it really
simple.

ADD is removed.

> GETDEVICELIST, but I'll also say up front that as Dave suggested we 
> may be able to handle the requirements via a new attribute.
> 
> * I do not see a need to register for notifications on a per-file 
> system basis.  I think the client can do this later via GETDEVICEINFO.
> Using
> GETDEVICELIST with the rootfh allows a client to register or 
> deregister globally.  The notifications arguments on a GETDEVICELIST 
> for a specific filesystem would be ignored by the server.  The client 
> can continue to register for notifications via GETDEVICEINFO.
> 

> but I missed (mea culpa) that we were removing the ability to query on

> a per filesystem basics for id to address mapping information.
> 
> > I don't think the server needs to be implemented to see if the 
> > entire global list is read in order to register ADD notifications. 
> > That would be onerous. I didn't see where I wrote that.
> 
> I misunderstood.
> 
> > in order to see if -- at the time the client mounts a fsid -- there 
> > is connectivity to the storage devices.
> 

> to communicate one bit of information to the client.  That doesn't 
> seem plausible to me.
> 
> I wouldn't think that per-fs GDL would be a requirement either, if 
> this were the goal.  Maybe a new per-fs attribute would be asked for.
> 

> set for which he has indicated he wants notification, but specifying 
> the notifications when doing a per-fs GDL and then not turning off 
> notifications for that fs via a per-fs GDL to turn off notifications 
> for that fs.  No ESP required.
> 
> I thought the issue was that there were clients that only wanted to 
> know about devices for a set of mounted fs's rather than for all 
> devices.
> I

> > How does the server know what fsids the client will be interested 
> > in?
> 
> Definitely agree.  We need more information about why they want this 
> and how they would use it, in order to figure out what notification 
> support, if any, is required.
> 
> 
> 
> 
> 
> 

> value in lookup by fsid in order to see if -- at the time the client 
> these questions come up. I think the simplest is that the last GDL or 
> GDI wins.
>  
> literally years.
> Until you include an array of fsids in the ADD notification, I don't 
> see what problem is being solved.
>

---

From: [email-masked]
Date: Sat, 19 Jan 2008 22:57:53 -0500

Dave,

I decline to offer an argument that per-fs notifications are needed.

Specifying that notifications can only be set on a global basis when using GDL is an acceptable simplification at the cost of clients receiving notifications that do not interest them.

In the world of block layouts there will not be frequent deletions or changes.  The cost of of receiving irrelevant notifications will be low.

Object layouts will not use GDL, so there is no issue there.

I suspect that this is an acceptable compromise for the files layout too, but defer to those more involved with that layout type.

-Jason

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Friday, January 18, 2008 3:12 PM
To: [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

ADD is removed.

---

From: [email-masked]
Date: Tue, 22 Jan 2008 09:58:16 -0500

Dave,

> > DELETE and CHANGE notifications can be set per-fsid subject 

Patient: "Doctor it hurts when I do <this>."
Doctor: "Don't do <that>!!"

At least for the block layout, we can ensure that devices aren't
shared among filesystems - if some physical device X is shared,
fs B reports the same device address information for device X as
fs A, but with a different device id.  Presto, no more notification
confusion.  I think this technique also works for the file layout.

> consequences of a simpler model, or maybe both.

Right - not sharing device ids across filesystems is a simpler
solution.  One can still share the actual devices by assigning
each device a different id for each filesystem and reporting the
same address info for each id.  This should get us back to the
notion of the fsid serving as a mask for the one true global
device list.

Thanks,
--David
 ---------------------------------------------------
David L. Black, Distinguished Engineer
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

> Sent: Saturday, January 19, 2008 8:10 AM
> > DELETE and CHANGE notifications can be set per-fsid subject 
> > > > appropriate DELETE and CHANGE notification for the set 
> of devices
> > it
> > 
> > > > had gotten-info about.
> > > 
> > > giving the client an error seems to much.  Also, how do 
> you get out
> > of
> > 
> don't expect 
> last GDL or 
> > > >    fs's for which ADDITION notification has been 
> specified.  When
> > a
> > > 
> > > devices) hen that is what you want to get ADDITION events 
> for: when
> > a
> > > >    in the interest of limiting storage use, not keep 
> precise note
> > > of

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 23 Jan 2008 11:18:52 -0500

> Doctor: "Don't do <that>!!" 

Exactly so.  Now we have to decide what "<this>" and "<that>" are.

Is it "trying to do per-fs notifications (when we have fs's sharing
device ids)" or is it "trying to have fs's sharing device id's (when we
have per-fs device notifications)"?

So it is clear that, it as at least very complicated to have both of
these in the same protocol.  I have sketched out how it might be done,
if both of these were absolutely required, but it is complicated and
certainly a bigger change than we would like to make at this point.

We have not heard anybody stand up and say that per-fs notifications are
required.  From what I understand, the impetus for this was more for
uniformity, to make per-fs GDL, match a possible global GDL.  So far,
nobody has explained why they simply must have per-fs notifications.

Let me explain why I would be at least very reluctant to dispense with
fs's sharing device id's.  First of all, although I can't speak for
others using a file layout type, it seems to me that the most natural
way to implement device management is to share device id's among large,
potentially quite large, numbers of fs's.  You might go all the way to
have device id's essentially denote servers but even if you don't, large
numbers of fs's will often be mapped to a single set of disks and with a
file layout the individual disks don't matter and the natural device id
would be to a large set of disks managed by a single server (an
"aggregate" in ONTAP terms).  It would be very common and reasonable to
have large numbers of fs's sharing devices in that sense.  That means
that saying that a device id can only be used for a single fs would mean
that where a client is accessing large numbers of fs's, it would have a
very large number of device id's to manage, and only because of what
appears a rather artificial restriction, in that if fs's were able to
share device id's, the number of devices to deal with might be quite
modest.  If there is a change or deletion of one of those devices, the
client winds up getting lots of notifications for what is essentially
the same device change or deletion, each under a different device id,
for each fs using that device.  We have to choose between two sorts of
unwanted notifications: changes to devices that are not being used by
your fs, and changes to devices that are being used by more than one fs.
Both are likely to be small on average, but the latter seems likely to
be much worse on a worst-case basis and I would worry about that.

The second reason is that this would be a significant architectural
change and the thing about architectural changes is that they sometime
have unanticipated consequences.  Therefore it is best to avoid them at
a late stage in construction, unless that is unavoidable.  In the
material world, the result may be unexplained structural issues showing
up years later and a suit against your architectual firm inching its way
through discovery.  With protocols, you can have email arguments about
who was at fault for some troublesome aspect of the protocol that never
seem to end.

Although I wasn't there, I understand that the issue of per-fs device
id's was discussed and decided at the session at the last IETF meeting.
That certainly can be re-opened if there is a good reason but we need to
have to good reason for us to conclude that the decision was mistaken.

> This should get us back to the notion of the fsid 
> serving as a mask for the one true global device list.

Obviously you like that model better than the current protocol and the
issues have had considerable discussion but if we are to make progress
here, we all have to accept that not everybody can get their first
choice on matters like this.  We want something that is workable for
everybody, and will be a Proposed Standard and implemented in our
lifetimes.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, January 22, 2008 9:58 AM
To: Noveck, Dave; [email-masked]; [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

Dave,

> > DELETE and CHANGE notifications can be set per-fsid subject to the 

At least for the block layout, we can ensure that devices aren't shared
among filesystems - if some physical device X is shared, fs B reports
the same device address information for device X as fs A, but with a
different device id.  Presto, no more notification confusion.  I think
this technique also works for the file layout.

> consequences of a simpler model, or maybe both.

Right - not sharing device ids across filesystems is a simpler solution.
One can still share the actual devices by assigning each device a
different id for each filesystem and reporting the same address info for
each id.  This should get us back to the notion of the fsid serving as a
mask for the one true global device list.

Thanks,
--David

> > DELETE and CHANGE notifications can be set per-fsid subject to the 
> Also note that if you turn on notifications for a device using GDL for

> a given fs and turn them off using GDL for that same fs, you may still

> have notifications set since those sets may be different and you are 
> not getting notifications when given devices more outside the set for 
> a specific fs, only when they are deleted completely or the mapping 
> changes.
> 

> done that.  The real issue is why/whether you need per-fs 
> notifications and so far, while I have heard explanations of why 
> per-fs GDL is needed, I haven't heard any of why per-fs notifications 
> are needed.
> If they are
> not needed, I think we should not put them, because taking that step 
> is where the jump in complexity lies.
>  
>

---

From: Mike Eisler <[email-masked]>
Date: Wed, 23 Jan 2008 08:43:25 -0800 (PST)

I don't think we are converging on a solution. I propose a 
conference call to be held this Friday, January 25, at NOON 
Pacific North American time, (3pm Eastern North American time), 
hosted by me.

Folks who plan to call in, or have a proxy call in should respond
*directly* to me. On Thursday, January 24, I will determine if there
will
be sufficient attendance, and if so, provide dial in details to
all participants.

I realize it is short notice and many likely have conflicting plans.
I suggest that this is an important issue, and perhaps your conflicts
can be re-scheduled. 

    -mre

---

From: [email-masked]
Date: Wed, 23 Jan 2008 16:35:31 -0500

Dave,

We may be close.  Selected excerpts and comments follow:

> device ids)" or is it "trying to have fs's sharing device 
> id's (when we have per-fs device notifications)"?
> 
> So it is clear that, it as at least very complicated to have both
> of these in the same protocol.

We agree on this.

> required.

I've seen Marc Eshel express a preference for per-fs notifications,
which he'll need to explain.  The block layout does not need them.

[... snip ...]

> fs's sharing device id's.

Let's see if we can agree on this summary restatement:

 	If per-fs notifications are wanted
	then device id's must not span fs's.

If the file layout can live with notifications being per-device-id,
the way forward is clear.  

[... snip ...]

> If there is a change or deletion of one of those devices, the

In other words, the device has a single instance state that
applies to *every* fs that uses the device.  That's an important
simplifying assertion that I hope we can agree upon as it leads
to ...

... if we agree, then I think Mike's proposal from last Friday

> DELETE and CHANGE are per-deviceID so nothing needs to change.

This is a consequence of device state being independent of fs.

> GETDEVICELIST can be made only per fsid.

Matches what the clients seem to want.

> DELETE and CHANGE notifications can be set per-fsid subject to
> the caveat that if a deviceID is shared by two fsids, and
> different notifications are set for the deviceID, last one wins.

I would think that a typical client is going to want the same
notification behavior across all fsids, so setting different
notifications for the same deviceID is an instance of a client
trying to be entirely too clever and "last one wins" is a fine
way to give that client what it deserves.

Making it clear that a device id has a single instance of state
that is independent of fs when the device is shared by multiple
fs's would be a good thing to do in the spec, and (IMHO) it
leads naturally to there being a single instance of notification
state for that device id for this client independent of fsid.

That still leaves (at least) one complication - from Jason's
message of last Friday:

> Consider the case of a client mounting multiple file systems
> that share some device ids.  What should such a client do when
> one file system is unmounted in order to disable future notifications?
> Sending a GETDEVICELIST message to disable the CHANGE notifications
> will inadvertently cancel the notifications for the devices (shared
> with another mounted file system) whose mappings are still cached.
> This is almost certainly not the desired behavior.
> 

Those sound workable, and the client is probably in a better position
to deal with the collection of fs's it still has mounted.

> discard all its cached information about the deviceid to device
address
> mapping for the devices used by the unmounted file system.

[... snip ...]

In this case the client is being lazy (that's ok).  A consequence
is that it may receive notifications for the devices that are only
used by the unmounted filesystem and need to discard those
notifications.  I don't see this as a problem.

Thanks,
--David

> Sent: Wednesday, January 23, 2008 11:19 AM
> To: Black, David; [email-masked]; [email-masked]
> > > > > applied to the specific device gotten-info and I 
> don't see what
> > > about
> > > > > per-fs GDL would change that.  In particular, a 
> reasonable model
> > > for
> > > 
> > > > > unknown devices it encounters and it would be nice if 
> the client
> > > got
> > > 
> interested 
> > > > Also, in this proposal, the client may be doing a 
> global GDL just
> > > to
> > > > This means that if I get a list of devices for fs A, 
> then there is
> > > no
> > > > way to be notified to additions to that set, if the 
> device is not
> > > new
> > > > >    It does not report additions to specific subsets 
> of that set
> > > of
> > > > >    interest.
> > > > 
> > > > >    global list and there are no notifications 
> regarding changes
> > > in
> > > > >    list, the set of interest consists of the set of 
> devices for
> > > all
> > > > the
> > > > If you are not interested in all devices but only those 
> for a set
> > > of
> > > > >    Similar principles apply to DELETE and CHANGE 
> notifications,
> > > but
> > > > >    above, which we will call the set of GDL-interest 
> and the set
> > > of
> > > > Assuming ADDITION does not apply to GDI notifications, 
> just as you
> > > do.
> > > > 
> > > > >    The notification contains no indication as to 
> which of these
> > > (it
> > > > >    could be both) applies.
> > > > 
> > > > >    In managing the sets GDI-interest and GDL-interest 
> a server,
> > > may
> > > > >    A deletion notification occurs when a device which 
> was in the
> > > > set
> > > > >    of GDL-interest leaves that set, and the device is 
> not in the
> > > > set
> > > > >    ceases to exist.  The notification contains no 
> indication of
> > > > which
> > > > >    of these is the case.
> > > > 
> > > > This sounds complicated.  That's because it is 
> complicated :-) and
> > > I

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 23 Jan 2008 18:14:26 -0500

> I would think that a typical client is going to want 
> the same notification behavior across all fsids, so 
> setting different notifications for the same deviceID 
> is an instance of a client trying to be entirely too 
> clever and "last one wins" is a fine way to give that 
> client what it deserves.

I think the typical client will want the same notification behavior
among all fsids that it has mounted.  I was assuming that the client
would not want notifications for fsids that didn't have mounted.  I'm
not going to touch the question of what such a client "deserves", but
I'm worried that this is a typical pattern of behavior which is going to
cause confusion.

Note that if the client does not turn off notifications for fsids that
he doesn't have mounted then that amounts to something like global
notifications and so I wonder what the value of per-fs notifications is.

I agree that we should give Marc an opportunity to explain why he wants
per-fs notifications.  Rather than more email exchanges, I think the
best way to do that is to talk this out at the conference call that Mike
has proposed.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Wednesday, January 23, 2008 4:36 PM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Changes to GETDEVICELIST in NFSv41 I-D 15 and 18

Dave,

We may be close.  Selected excerpts and comments follow:

> device ids)" or is it "trying to have fs's sharing device id's (when 
> we have per-fs device notifications)"?
> 
> these in the same protocol.

We agree on this.

[... snip ...]

If the file layout can live with notifications being per-device-id, the
way forward is clear.  

[... snip ...]

> If there is a change or deletion of one of those devices, the client 
> winds up getting lots of notifications for what is essentially the 
> same device change or deletion, each under a different device id,

In other words, the device has a single instance state that applies to
*every* fs that uses the device.  That's an important simplifying
assertion that I hope we can agree upon as it leads to ...

... if we agree, then I think Mike's proposal from last Friday will

I would think that a typical client is going to want the same
notification behavior across all fsids, so setting different
notifications for the same deviceID is an instance of a client trying to
be entirely too clever and "last one wins" is a fine way to give that
client what it deserves.

Making it clear that a device id has a single instance of state that is
independent of fs when the device is shared by multiple fs's would be a
good thing to do in the spec, and (IMHO) it leads naturally to there
being a single instance of notification state for that device id for
this client independent of fsid.

That still leaves (at least) one complication - from Jason's message of
last Friday:

> Consider the case of a client mounting multiple file systems that 
> share some device ids.  What should such a client do when one file 
> system is unmounted in order to disable future notifications?

> cancel the notifications for those devices that are no longer needed.

Those sound workable, and the client is probably in a better position to
deal with the collection of fs's it still has mounted.

address

[... snip ...]

In this case the client is being lazy (that's ok).  A consequence is
that it may receive notifications for the devices that are only used by
the unmounted filesystem and need to discard those notifications.  I
don't see this as a problem.

Thanks,
--David

> device ids)" or is it "trying to have fs's sharing device id's (when 
> we have per-fs device notifications)"?
> 

> We have not heard anybody stand up and say that per-fs notifications 
> are required.  From what I understand, the impetus for this was more 
> for uniformity, to make per-fs GDL, match a possible global GDL.  So 
> far, nobody has explained why they simply must have per-fs 
> notifications.
> 

> way to implement device management is to share device id's among 
> large, potentially quite large, numbers of fs's.  You might go all the

> way to have device id's essentially denote servers but even if you 
> don't, large numbers of fs's will often be mapped to a single set of 
> disks and with a file layout the individual disks don't matter and the

> natural device id would be to a large set of disks managed by a single

> server (an "aggregate" in ONTAP terms).  It would be very common and 
> reasonable to have large numbers of fs's sharing devices in that 
> sense.  That means that saying that a device id can only be used for a

> single fs would mean that where a client is accessing large numbers of

> fs's, it would have a very large number of device id's to manage, and 
> only because of what appears a rather artificial restriction, in that 
> if fs's were able to share device id's, the number of devices to deal 
> with might be quite modest.  If there is a change or deletion of one 
> of those devices, the client winds up getting lots of notifications 
> for what is essentially the same device change or deletion, each under

> a different device id, for each fs using that device.  We have to 
> choose between two sorts of unwanted notifications: changes to devices

> that are not being used by your fs, and changes to devices that are 
> being used by more than one fs.

> have unanticipated consequences.  Therefore it is best to avoid them 
> at a late stage in construction, unless that is unavoidable.  In the 
> material world, the result may be unexplained structural issues 
> showing up years later and a suit against your architectual firm 
> inching its way through discovery.  With protocols, you can have email

> arguments about who was at fault for some troublesome aspect of the 
> protocol that never seem to end.
> 
> That certainly can be re-opened if there is a good reason but we need 
> to have to good reason for us to conclude that the decision was 
> mistaken.
> 
> > This should get us back to the notion of the fsid serving as a mask 
> > for the one true global device list.
> 

> shared among filesystems - if some physical device X is shared, fs B 
> reports the same device address information for device X as fs A, but 
> with a different device id.  Presto, no more notification confusion.  
> I think this technique also works for the file layout.
> 
> solution.
> different id for each filesystem and reporting the same address info 
> for each id.  This should get us back to the notion of the fsid 
> David L. Black, Distinguished Engineer EMC Corporation, 176 South St.,

> Hopkinton, MA  01748

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 24 Jan 2008 09:11:35 -0600

Marc,

Your name has come up a couple of times in this thread with respect
to requirements.  Please respond fully so that we can determine what
the true requirements are and can move forward with this issue.

Spencer

On Jan 23, 2008, at 5:14 PM, Noveck, Dave wrote:

> are
> trying to
> that is
> message of

---

From: Marc Eshel <[email-masked]>
Date: Thu, 24 Jan 2008 13:44:58 -0500

I though that we have a confcall tomorrow to resolve this issue. Mike, Is 
the call still on ?
In general the implementation on the Linux server side is delegating most 
of the device/layout management to the underlaying file system so we 
prefer that unit of management will be per file system.
Marc.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 24 Jan 2008 12:46:41 -0600

Yes, we are having a conference call.  I want to see the requirements
enumerated in email from you and others that have similar requirements
such that they are captured in the archives; the resolution can be
discussed in the concall; requirements beforehand, please.

Spencer

On Jan 24, 2008, at 12:44 PM, Marc Eshel wrote:

>>> applies to
>>>> fs B

---

From: Mike Eisler <[email-masked]>
Date: Thu, 24 Jan 2008 10:50:36 -0800 (PST)

> I though that we have a confcall tomorrow to resolve this issue.

The call is on. When responses stop coming in, I will send dial info
to each respondent today.

---

From: [email-masked]
Date: Thu, 24 Jan 2008 16:27:42 -0500

Mike,

Will the audio-call to discuss the changes to GETDEVICELIST take place
tomorrow ?

If yes, could you please send us the dial-in information ?

Thank you
Cheers
Mario

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Wednesday, January 23, 2008 11:43 AM
To: [email-masked]
Subject: Conference Call Proposed (was RE: [nfsv4] Changes to
GETDEVICELIST inNFSv41 I-D 15 and 18)

    -mre