Subject: Re: [nfsv4] Fwd: New Version Notification for draft-bhalevy-nfs-obj-00.txt

From: "J. Bruce Fields" <[email-masked]>
Date: Fri, 7 Sep 2012 11:50:07 -0400

On Fri, Aug 31, 2012 at 04:17:58PM +0300, Benny Halevy wrote:
> I've submitted a proposed extension to the pnfs objects layout
> that introduces the use of NFS filers as object storage data servers.
> 
> The following features are proposed as well:
> * OSD Multi-path: The OSD address may provide an array of network addresses (ooa_netaddrs).
> * I/O Statistics: The client may report I/O stats on LAYOUTRETURN
> 
> The motivation for adding support for NFS comes from:
> 1. The desire to use an ubiquitous, standard protocol to access the data servers.
> The T10 OSD protocol, although standard, lacks wide adoption, while NFS and in particular
> NFSv3 is very popular and widely available.
> 
> 2. Encourage best-of-breed solution.
> In contrast to the files layout, the object layout does not require a proprietary
> back-end protocol, hence the proposed layout allows one to mix and match a metadata
> server and data servers from different vendors.

That sounds interesting.

> A simple security control method is proposed to achieve that:
> The metadata server controls the file ownership and permissions of the objects
> stored on data servers and the client is handed a corresponding RPC credential
> on LAYOUTGET a-la OSD capabilities.  Outstanding credentials are unilaterally
> revoked by the MDS by modifying the objects user or group owner.

Is that also the way you enforce mandatory byte-range locks and shares?
In that case you need a nearly infinite supply of owners, don't you?
I'm curious how that will work.

Miscellaneous nits and questions:

Throughout xdr fields are often defined as opaque objects whose contents
are encoded like some type from rfc 5661 (see e.g. nid_fhandle,
onc_auth.)  Is it not possible to just use the original types directly?

Section 3.4 gives a lot of details on security requirements for
capabilities in the OSD case.  Shouldn't something be said about the NFS
case?

Section 4.2 allows using either netaddr's or fqdn's to address data
servers.  The file layout gets away with just netaddr's.  Why should
this be different?

In the same pnfs_obj_nfs_add4: what will the client use ona_path for?
It addresses files by filehandle, not path.

End of section 6.2, "better not" is a bit colloquial, maybe "can't write
new attributes accurately until...".  Also: why the recommendation that
LAYOUTCOMMIT and LAYOUTRETURN occur in the same compound?

Section 8.1: error descriptions refer to "the IO operation" everywhere,
but normally the client will perform lots of operations with a layout,
and may encounter multiple errors.  Is it required to include one
olr_ioerr_report for every failed IO?  Also: what use can the server
make of PNFS_OBJ_ERR_RESOURCE, and why is the MUST in that description
important?

Section 9, "defined in the NFSv4.1" should be "defined in NFSv4.1".

Section 12, the words "at the least" can be omitted.

Section 13 begins assuming (mostly?) the OSD case, with no mention of
the NFS case until 13.5.  This is confusing.  It would be clearer if
there were at least some mention of the NFS case in the introduction to
that section, and if it were made explicit which parts were
OSD-specific.  Maybe it should be structured like

	13. Security considerations
		- generic introduction
	
	13.1. Security considerations over OSD protocol
		- subsections here
	13.2. Security considerations over NFS

--b.

---

From: Benny Halevy <[email-masked]>
Date: Tue, 11 Sep 2012 22:24:00 +0300

Hi Bruce,

Thanks a lot for your review!
my answers are in line below

Benny

> I'm curious how that will work.

The plan is to enforce mandatory byte range locks on the MDS by recalling layouts
conflicting with the locked regions.

> onc_auth.)  Is it not possible to just use the original types directly?

specifically for the filehandle, it may differ between different DS NFS versions.

> capabilities in the OSD case.  Shouldn't something be said about the NFS

Makes sense.

> servers.  The file layout gets away with just netaddr's.  Why should

fqdn's are an easy path to get dynamic resolving and multi-pathing to the DS
using just ip addresses presents a problem if these are not static and might change.
This will require the server to update the device info, say be sending a CB_NOTIFY_DEVICEID.

> It addresses files by filehandle, not path.

ona_path is the mount path on the DS.
It's required for the client to mount the DS and to optionally validate that it's
accessing the right DS (if the server makes sure this path is unique in the cluster)

> LAYOUTCOMMIT and LAYOUTRETURN occur in the same compound?

This was carried forward from rfc5664.
I see no problem in clarifying the text.
the recommendation applies to servers that store file attributes on
the DS's and since the error report arrives on LAYOUTRETURN, knowing
that there were errors at LAYOUTCOMMIT time, could help avoid getting
and handling errors when writing the updated attributes to the DS.

> olr_ioerr_report for every failed IO?

no, the client aggregates the errors, but as you pointed out
the spec. might be unclear about that.  How about saying referring
to the operations in plural?

> Also: what use can the server

With RAID writes, when multiple errors are encountered and some are permanent,
like device or unrecoverable media failures, the affected stripes may
become unrecoverable.  Knowing that writes were not issued to storage
can save these stripes from this unfortunate fate.

> Section 9, "defined in the NFSv4.1" should be "defined in NFSv4.1".

thanks

> Section 12, the words "at the least" can be omitted.

thanks

> 	13.2. Security considerations over NFS

makes a lot of sense.
I'll incorporate this into the next version
and will post up for review.

Benny

> 
> --b.
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 13 Sep 2012 09:38:44 -0400

> accessing the right DS (if the server makes sure this path is unique in the cluster)

So this is purely a sanity check in case the data servers were
misconfigured?

I'm a little skeptical of this idea.

If you're set on doing this, at a minimum I think you'd need to define
what you mean by "mount".

(And: I don't recall seeing this explained in the spec.  But there's a
good chance I just overlooked it.)

--b.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 17 Sep 2012 16:34:35 +0300

> So this is purely a sanity check in case the data servers were

Or in case the data server's ip address is dynamic and changed.

> what you mean by "mount".

For NFSv3 as the data access protocol, mount means using the MOUNT protocol
to get access to the DS. That said, I can envision a namespace-less DS that
does not export any paths, and since mounting the DS is not mandatory for
accessing via filehandles I'm willing to change the server requirement for
ona_path to "SHOULD provide" rather than "MUST provide."

> good chance I just overlooked it.)

No, I think you're right on the spot.

Thanks,

Benny