Subject: [nfsv4] What is the point of GETDEVICELIST?

From: Christoph Hellwig <[email-masked]>
Date: Tue, 26 Nov 2013 16:55:51 +0100

Section 12.2.10. of RFC5661 says:

"The device ID (data type deviceid4, see Section 3.3.14) identifies a
 group of storage devices.  The scope of a device ID is the pair
 <client ID, layout type>."

and

"A device ID lives as long as there is a layout referring to the
 device ID.  If there are no layouts referring to the device ID, the
 server is free to delete the device ID any time."

While section 18.41.3. which describes GETDEVICELIST says:

" This operation is used by the client to enumerate all of the device
  IDs that a server's file system uses. The client provides a current
  filehandle of a file object that belongs to the file system (i.e.,
  all file objects sharing the same fsid as that of the current
  filehandle) and the layout type in gdia_layout_type."

Given the description of how a deviceid works the call seems fairly
useless as will just provide a racy snapshot.  Also the suggestion

"An example of the use of this operation is for pNFS clients and
 servers that use LAYOUT4_BLOCK_VOLUME layouts.  In these environments
 it may be helpful for a client to determine device accessibility upon
 first file system access."

Seems odd.  Block volumes don't behave different from NFSv⒋.1 data
server or especially T10 OSD in that respect.  In fact the T10 OSD
are generally enumerated in exactly the same way as SCSI stacks in
all OSes I know.

Unfortunately it seems like the authers of the Linux block layout
client followed that misguided advice and thus created and implementation
that fails to adhere to the constraints mentioned in 12.2.10., in that
it won't work if device ids are only created on demand when performing
a LAYOUTGET.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 26 Nov 2013 18:13:38 +0200

> useless as will just provide a racy snapshot.  Also the suggestion

That's a good point.

> are generally enumerated in exactly the same way as SCSI stacks in

The main difference is in GETDEVICEINFO where the object layout provides
enough addressing information to dynamically discover the device and
connect to it while the block layout provides just a on-disk block signature
that the client needs to look for over all block devices the admin already
configured for the client (before mounting the NFS server).

Benny

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: Christoph Hellwig <[email-masked]>
Date: Tue, 26 Nov 2013 17:17:44 +0100

On Tue, Nov 26, 2013 at 11:12:09AM -0500, Fred Isaman wrote:
> My memory is that finding the disk associated with a particular device id
> was very expensive, and this was an attempt to front load that cost to
> mount.

Indeed the way GETDEVICEINFO is specified for block layouts with the overly
generic signature basically requires a linear scan.  If it had identified
a fixed-offset UUID it could have relied on generic probing and caching
layers like libblkid in Linux that makes mounts by label or uuid to have
much less overhead already.

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 03:31:35 +0000

>-----Original Message-----
>From: nfsv4 [mailto:[email-masked]] On Behalf Of Christoph Hellwig
>Sent: Tuesday, November 26, 2013 11:56 PM
>To: [email-masked]; Black, David; Fred Isaman
>Subject: [nfsv4] What is the point of GETDEVICELIST?
>
>"The device ID (data type deviceid4, see Section 3.3.14) identifies a  group of
>storage devices.  The scope of a device ID is the pair  <client ID, layout type>."
>
>and
>
>"A device ID lives as long as there is a layout referring to the  device ID.  If
>there are no layouts referring to the device ID, the  server is free to delete the
>device ID any time."
>
>Given the description of how a deviceid works the call seems fairly useless as
>will just provide a racy snapshot.  Also the suggestion
>
If device ID does change at server side, client can register for CB_NOTIFY_DEVICEID. Then server is required to send CB_NOTIFY_DEVICEID when device ID changes and client can then update its local cached mapping. Please see section 22.2 of rfc5661.

>"An example of the use of this operation is for pNFS clients and  servers that
>use LAYOUT4_BLOCK_VOLUME layouts.  In these environments  it may be
>helpful for a client to determine device accessibility upon  first file system
>access."
>
>Seems odd.  Block volumes don't behave different from NFSv⒋.1 data server
>or especially T10 OSD in that respect.  In fact the T10 OSD are generally
>enumerated in exactly the same way as SCSI stacks in all OSes I know.
>
Yes you are right. IMO, GETDEVICELIST is more of an optimization for client to possibly implementing a device id cache that can avoid device probing at IO time.

>Unfortunately it seems like the authers of the Linux block layout client
>followed that misguided advice and thus created and implementation that
>fails to adhere to the constraints mentioned in 12.2.10., in that it won't work if
>device ids are only created on demand when performing a LAYOUTGET.
Device ids are of scope <client ID, layout type>. Server is fully capable of creating device IDs when client mounts. If server wants to change device IDs later on, it needs to implement CB_NOTIFY_DEVICEID.

For Linux block layout client, although CB_NOTIFY_DEVICEID is implemented at generic layer, it doesn't support dynamically identify the new mapping if a new device ID is found during layoutget. So that needs to be fixed if we ever want to work with a server that changes device IDs.

OTOH, what is the benefit of creating device IDs on demand during LAYOUTGET? One stripe pattern for each layout? That sound a bit crazy at least for a block layout server, where device IDs usually identifies volume topology.

Thanks,
Tao

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 03:33:41 +0000

>Sent: Wednesday, November 27, 2013 12:18 AM
>To: Fred Isaman
>Cc: [email-masked]; Black, David
>Subject: Re: [nfsv4] What is the point of GETDEVICELIST?
>
>> My memory is that finding the disk associated with a particular device
>> id was very expensive, and this was an attempt to front load that cost
>> to mount.
>
>generic signature basically requires a linear scan.  If it had identified a fixed-
>offset UUID it could have relied on generic probing and caching layers like
>libblkid in Linux that makes mounts by label or uuid to have much less
>overhead already.
At least for EMC servers, we did store the signature at different fixed offset depending on server version. I'm not sure if libblkid can identify that...

Thanks,
Tao

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 09:16:28 +0100

> configured for the client (before mounting the NFS server).

That's true.  What is the historic reasons for these arbtirary differences?

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 09:22:23 +0100

> If device ID does change at server side, client can register for CB_NOTIFY_DEVICEID. Then server is required to send CB_NOTIFY_DEVICEID when device ID changes and client can then update its local c…

That is only one of two options according to 12.2.10 in RFC5661:

"A server has two choices for changing mappings.  It can recall all layouts 
 referring to the device ID or it can use a notification mechanism."

The device ID notifcations are marked optional in the spec, so a server
is perfectly fine to remove and add device IDs as long as no outstanding
layouts refer to them, either because they were never handed out or already
recalled.

> Device ids are of scope <client ID, layout type>. Server is fully capable of creating device IDs when client mounts. If server wants to change device IDs later on, it needs to implement CB_NOTIFY_D…

It might be capable, but as-is the spec does not require it to do so

> OTOH, what is the benefit of creating device IDs on demand during LAYOUTGET? One stripe pattern for each layout? That sound a bit crazy at least for a block layout server, where device IDs usually…

I'm not talking about practical implementation here, just language lawyering
the spec as I wade through it.

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 09:23:42 +0100

> At least for EMC servers, we did store the signature at different fixed offset depending on server version. I'm not sure if libblkid can identify that...

If you publish them it easily could, but it rather defeats the point of
having a standard specified if it requires vendor and even version specific
hacks to work efficient.

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 09:20:28 +0000

>Sent: Wednesday, November 27, 2013 4:22 PM
>To: Peng, Tao
>Cc: [email-masked]; Fred Isaman
>> If device ID does change at server side, client can register for
>CB_NOTIFY_DEVICEID. Then server is required to send CB_NOTIFY_DEVICEID
>when device ID changes and client can then update its local cached mapping.
>Please see section 22.2 of rfc5661.
>
>The device ID notifcations are marked optional in the spec, so a server is
>perfectly fine to remove and add device IDs as long as no outstanding layouts
>refer to them, either because they were never handed out or already recalled.
>
Yeah, that does sound like a protocol defect. If the device IDs returned by GETDEVICELIST cannot be trusted, the existence of such a call is meaningless.

Implementation wise, all three layout clients in Linux kernel seem to fail to comply the requirement. File and object layouts cache device IDs until CB_NOTIFY_DEVICEID. Block layout just cache it forever. So I guess we just haven't seen a server that changes device ID mapping yet.

Thanks,
Tao

>> Device ids are of scope <client ID, layout type>. Server is fully capable of
>creating device IDs when client mounts. If server wants to change device IDs
>later on, it needs to implement CB_NOTIFY_DEVICEID.
>
>> OTOH, what is the benefit of creating device IDs on demand during
>LAYOUTGET? One stripe pattern for each layout? That sound a bit crazy at
>least for a block layout server, where device IDs usually identifies volume
>topology.
>

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 09:25:20 +0000

>Sent: Wednesday, November 27, 2013 4:24 PM
>To: Peng, Tao
>Cc: Christoph Hellwig; [email-masked]; Fred Isaman
>> >Indeed the way GETDEVICEINFO is specified for block layouts with the
>> >overly generic signature basically requires a linear scan.  If it had
>> >identified a fixed- offset UUID it could have relied on generic
>> >probing and caching layers like libblkid in Linux that makes mounts
>> >by label or uuid to have much less overhead already.
>> At least for EMC servers, we did store the signature at different fixed offset
>depending on server version. I'm not sure if libblkid can identify that...
>
>If you publish them it easily could, but it rather defeats the point of having a
>standard specified if it requires vendor and even version specific hacks to
>work efficient.
Yeah, that's why block layout protocol gives the control to servers about where to find the signature, because it might just vary with different vendors, and it is improper for a network protocol to define server file system's on-disk layout.

Thanks,
Tao

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 10:44:21 +0100

> Yeah, that's why block layout protocol gives the control to servers about where to find the signature, because it might just vary with different vendors, and it is improper for a network protocol t…

I think it's perfectly fine to define the signature at a specific offset,
as the server can shift it's actual filesystem data by an arbitrary amount
to sit entirely behind that offset.  RFC 6688 even defines a very specific
volume header for PNFS block volumes that is much more limiting to servers
than this would ever be.

The sad part is that the EFI GPT label referenced by that standard actually
defines a per-disk GUID and per-partition GUIDs that could have been
trivially used for that effect just by defining a
PNFS_BLOCK_VOLUME_EFI type:

struct pnfs_block_efi_volume_info4 {
	guid_t		volume_guid;
}

union pnfs_block_volume4 switch (pnfs_block_volume_type4 type) {
	...
	case PNFS_BLOCK_VOLUME_EFI:
		pnfs_block_eif_volume_info4
	..
}

and allowing to use it instead of the simple volume as the base
for the hierachy.

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 10:48:48 +0100

> Implementation wise, all three layout clients in Linux kernel seem to fail to comply the requirement. File and object layouts cache device IDs until CB_NOTIFY_DEVICEID. Block layout just cache it f…

12.2.10 also says:

"Once a device ID is
 deleted by the server, the server MUST NOT reuse the device ID for
 the same layout type and client ID again."

So keeping the old deviceids around is just the equivalent of a resource
leak on the client, but should not harm proper operation otherwise.
But the Linux block client never does a GETDEVICEINFO outside mount time,
so it's still broken if it ever receives a layout for a device that didn't
exist at mount time.

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 10:19:47 +0000

>Sent: Wednesday, November 27, 2013 5:49 PM
>To: Peng, Tao
>> Yeah, that does sound like a protocol defect. If the device IDs returned by
>GETDEVICELIST cannot be trusted, the existence of such a call is meaningless.
>>
>> Implementation wise, all three layout clients in Linux kernel seem to fail to
>comply the requirement. File and object layouts cache device IDs until
>CB_NOTIFY_DEVICEID. Block layout just cache it forever. So I guess we just
>haven't seen a server that changes device ID mapping yet.
>
>12.2.10 also says:
>
>"Once a device ID is
> deleted by the server, the server MUST NOT reuse the device ID for  the
>same layout type and client ID again."
>
>So keeping the old deviceids around is just the equivalent of a resource leak
>on the client, but should not harm proper operation otherwise.
>so it's still broken if it ever receives a layout for a device that didn't exist at
>mount time.
What I meant is that RFC5661 gives server the freedom to alter (instead of delete) device ID to device address mapping. In such case without NOTIFY_DEVICEID4_CHANGE, all three Linux layout clients fail to acknowledge the new mapping.

And I agree that Linux block layout client is more broken as it cannot work with device ID addition.

Thanks,
Tao

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 11:26:46 +0100

> What I meant is that RFC5661 gives server the freedom to alter (instead of delete) device ID to device address mapping. In such case without NOTIFY_DEVICEID4_CHANGE, all three Linux layout clients…

Yes.

That being said I think changing device ids on the fly is extremely dangerous,
and servers should always delete the old ID and add a new one instead to make
sure a sane error path exists for clients using outdata information.  Some
of the language in 12.2.10. seems to be written with that in mind, including
the explicit veto on reusing device IDs, but other bits of the spec seem
to have a different model in mind.

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 10:49:19 +0000

>From: Christoph Hellwig [mailto:[email-masked]]
>Sent: Wednesday, November 27, 2013 5:44 PM
>To: Peng, Tao
>> Yeah, that's why block layout protocol gives the control to servers about
>where to find the signature, because it might just vary with different vendors,
>and it is improper for a network protocol to define server file system's on-disk
>layout.
>
>I think it's perfectly fine to define the signature at a specific offset, as the
>server can shift it's actual filesystem data by an arbitrary amount to sit entirely
>behind that offset.  RFC 6688 even defines a very specific volume header for
>PNFS block volumes that is much more limiting to servers than this would ever
>be.
>
>defines a per-disk GUID and per-partition GUIDs that could have been trivially
>used for that effect just by defining a PNFS_BLOCK_VOLUME_EFI type:
>
>and allowing to use it instead of the simple volume as the base for the
>hierachy.
I don't see any problem with such a change. pnfs_block_eif_volume_info4 needs to be an array of EFI GPT labels btw. And it looks compatible with existing server implementation which returns other volume types. Maybe others can comment more.

Thanks,
Tao

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 10:56:48 +0000

>Sent: Wednesday, November 27, 2013 6:27 PM
>To: Peng, Tao
>> What I meant is that RFC5661 gives server the freedom to alter (instead of
>delete) device ID to device address mapping. In such case without
>NOTIFY_DEVICEID4_CHANGE, all three Linux layout clients fail to acknowledge
>the new mapping.
>
>Yes.
>
>and servers should always delete the old ID and add a new one instead to
>make sure a sane error path exists for clients using outdata information.
>Some of the language in 12.2.10. seems to be written with that in mind,
>including the explicit veto on reusing device IDs, but other bits of the spec
>seem to have a different model in mind.
Totally agree :)

Maybe forbidding device ID change on the fly in RFC5661 is better at the stage? Otherwise client has to invalidate device ID cache once it is on longer referenced after layoutreturn.

Thanks,
Tao

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 12:16:15 +0100

> I don't see any problem with such a change. pnfs_block_eif_volume_info4 needs to be an array of EFI GPT labels btw. And it looks compatible with existing server implementation which returns other v…

I don't see the need for an array, the CONCAT and STRIPE volume types
would remain as-is to build more complex devices.  The SLICE one could
also still exist inside an EFI partition, similar to how the Various PC
BSD systems long used their own slices inside of MSDOS partitions tables
(not sure if they still do, and if so also inside EFI GPT partitions).

---

From: "Peng, Tao" <[email-masked]>
Date: Wed, 27 Nov 2013 11:34:14 +0000

>Sent: Wednesday, November 27, 2013 7:16 PM
>To: Peng, Tao
>> I don't see any problem with such a change. pnfs_block_eif_volume_info4
>needs to be an array of EFI GPT labels btw. And it looks compatible with
>existing server implementation which returns other volume types. Maybe
>others can comment more.
>
>I don't see the need for an array, the CONCAT and STRIPE volume types would
>remain as-is to build more complex devices.  The SLICE one could also still exist
>inside an EFI partition, similar to how the Various PC BSD systems long used
>their own slices inside of MSDOS partitions tables (not sure if they still do, and
>if so also inside EFI GPT partitions).
>
You are right. pnfs_block_simple_volume_info4 uses array because server might want client to verify multiple signatures on the same device. EFI GPT label is per partition signature so it is not array. Sorry for the noise.

Thanks,
Tao

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 12:52:42 +0100

That brings up the question how we'd provide updates to the pnfs block
or object specifications given that they are separate RFCs?

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Nov 2013 17:25:16 +0200

> or object specifications given that they are separate RFCs?

These RFCs can be updated via the normal IETF process of errata,
bis draft, and updated RFCs that obsolete the old ones.
Cross-RFC versioning synchronization is a bit of a challenge but
we should get it right in the references section by referring to
drafts while they are so, that will also create the right dependencies
for the standardization / publication process and those place holders
are fixed upon submission by the IETF RFC editor.

Benny

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Nov 2013 17:28:11 +0200

> That's true.  What is the historic reasons for these arbtirary differences?

Mainly heritage from the then existing implementations.
My original motivation behind pnfs as a whole with this respect was to allow
zero administration on the client hosts so that the cluster administration would
be done centrally on the MDS and the client side dynamic configuration would
be done automatically via on-the-wire device info. That approach wasn't fully
bought-in by those who defined the block layout, I'm not sure exactly why...

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 16:44:17 +0100

> are fixed upon submission by the IETF RFC editor.

How does a server know a client speaks a new dialect of block or object
volumes?  We could define a new layouttype for it, but I don't think
that scales very well.  In fact the file layout seems to have the exact
same issue given it is specieid as a NFSv4.1 file layout and not for
any later versions.

---

From: Fred Isaman <[email-masked]>
Date: Tue, 26 Nov 2013 11:12:09 -0500

Fred

---

From: "faibish, sorin" <[email-masked]>
Date: Wed, 27 Nov 2013 11:06:05 -0500

Sent from my iPad

On Nov 27, 2013, at 11:04 AM, "Fred Isaman" <[email-masked]<mailto:[email-masked]>> wrote:

On Tue, Nov 26, 2013 at 10:55 AM, Christoph Hellwig <[email-masked]<mailto:[email-masked]>> wrote:
Section 12.2.10. of RFC5661 says:

and

[email-masked]<mailto:[email-masked]>

My memory is that finding the disk associated with a particular device id was very expensive, and this was an attempt to front load that cost to mount.

Yes. This was the reason.

./Sorin

Fred

_______________________________________________
nfsv4 mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Wed, 27 Nov 2013 18:14:01 +0200

> same issue given it is specieid as a NFSv4.1 file layout and not for

Right.  If we change the wire protocol we'll need to define a new layout type.
That's related to the who;e minor versioning discussion Dave Noveck started.
My position is to make the protocol modular such that the base protocol
(call it NFSv5 for this purpose) defines the common data structures and
mandatory operations that will allow the client to discover and negotiate
the server capabilities - optional features (or feature sets) and their
respective supported versions.

This will allow independent progress on various features in their own pace.
Cross-feature version dependencies need to be carefully documented and maybe
represented in the protocol itself.  With that in mind the layout type itself,
could be thought of a protocol extension that is versioned and depends on the
base protocol feature that is the generic pnfs set of operations.
This way you can updated the base feature version and modify the operations
and then respectively update the layout type and have them depend on the
new base version.

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 27 Nov 2013 16:34:49 +0000

On Nov 27, 2013, at 10:44 AM, Christoph Hellwig <[email-masked]> wrote:

>   In fact the file layout seems to have the exact

It is valid for all following minorversions until a minorversion states
that it is not.

---

From: "Haynes, Tom" <[email-masked]>
Date: Wed, 27 Nov 2013 16:42:58 +0000

On Nov 27, 2013, at 11:14 AM, Benny Halevy <[email-masked]<mailto:[email-masked]>> wrote:

There seems to be some sentiment that the negotiation process will be sweet
and have no issues at all.

I see it being just as big a nightmare:

1) Dependency graphs having cycles.
2) Internal qualification efforts for testing interoperability.

So before we dive deep into this new koolaid, can we get the requirements in
place and also see if there are issues to be dealt with?

---

From: Boaz Harrosh <[email-masked]>
Date: Wed, 27 Nov 2013 19:18:27 +0200

> or object specifications given that they are separate RFCs?

The object specifications is not relevant to this topic and *need not
be touched*. There is a standardized and well defined signature mechanism
for each osd LUN (Partitions are not relevant in osd they are not like
disk Partitions, though even Partitions have unique ID via osdname-attr)

These are done via osd-get-attribute commands. In Linux the osd stack
exports these signatures via sysfs on login.

Hey they must be well defined the signature and and security is based
on these?

There is very similar definition for disk-with-security that was copied
almost verbatim from t10-osd. That could be used as well, easily with
the such as iscsi-targets like LIO.

Also there is a "SYSTEM_ID" defined for DISK-TARGETS by T10 in the
Device Identification VPD page (see SPC-4):

So an SYSTEM_ID/Partition# could be used as a well defined blocks-device
unique ID.

Cheers
Boaz

---

From: Christoph Hellwig <[email-masked]>
Date: Wed, 27 Nov 2013 18:27:53 +0100

On Wed, Nov 27, 2013 at 07:18:27PM +0200, Boaz Harrosh wrote:
> be touched*.

No need to get a heat attack.  The discussion is about how to change it
_iff_ needed.  I'm pretty sure it'll get there eventually.

---

From: "Noveck, David" <[email-masked]>
Date: Thu, 28 Nov 2013 09:28:34 -0500

Ø  There seems to be some sentiment that the negotiation process will be sweet and have no issues at all.

Despite the word "negotiation" being used, I haven't seen any suggestion that anything that is really a negotiation is on the table.  What Benny is saying seems  pretty much compatible with feature discovery as it exists today.  The client finds out what set of optional features the server supports and then the client uses some subset of those as it chooses.

I've talked about feature discovery methods that are more efficient than trying sets of optional ops/enums but that streamlining doesn't change the basic model.

If Benny wants to change that basic model, he should write an I-D explaining how and why.

Ø  I see it being just as big a nightmare:

I see it (negotiation) as non-existent.  For now it is just a handwave and I expect it to remain that until somebody proposes OP_EXCHANGE_ID_AND_BITCOIN and clients and servers actually negotiate.  Great AI project, though :-)

NFSv5 though, I do see as a nightmare.

From: nfsv4 [mailto:[email-masked]] On Behalf Of Haynes, Tom
Sent: Wednesday, November 27, 2013 11:43 AM
To: Benny Halevy
Cc: Isaman, Fred; Peng, Tao; Christoph Hellwig; [email-masked]
Subject: Re: [nfsv4] What is the point of GETDEVICELIST?