Subject: Re: [nfsv4] NFSv4 WG Last Call for pNFS Flexible File Layout - closes on Jan 20th

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 1 Feb 2015 17:53:18 -0500 (EST)

On Sat, 17 Jan 2015, Trond Myklebust wrote:

> Hi Ben,
>
> On Thu, Jan 15, 2015 at 11:11 PM, Benjamin Kaduk <[email-masked]> wrote:
> > On Thu, 15 Jan 2015, Trond Myklebust wrote:
> >
> >> On Thu, Jan 15, 2015 at 10:00 PM, Benjamin Kaduk <[email-masked]> wrote:
> >> > I only got a chance to skim the document, but it looks like the metadata
> >> > server could use the services for user (S4U) kerberos extensions from
> >> > Microsoft [MS-SFU] to get an actual ticket and act on behalf of the real
> >> > client.  The client could then auth to the storage device with its normal
> >> > credentials (i.e., the user's kerberos ticket).  The metadata server would
> >> > need to be marked as having elevated privilege of one form or another on
> >> > the KDC, of course.
> >>
> >> You're missing the point. It is trivial to give the MDS elevated
> >
> > My apologies.
> >
> > Sorry for the noise,
>
> I didn't intend to shut you up, just to clarify what the goals are.

Understood.  Sorry for the delay in responding -- I wanted to actually go
and do a more thorough read-through of the document first.

> I'm hoping the above idea can work, but it would have to work to
> authenticate the client to a DS.

I'm still a little unclear on the why of this.  The security model (per
section 2.2) is that the metadata server uses synthetic uids and gids for
the data file -- is there something that prevents those "synthetic" uids
from being the real kerberos identity of the user which was given the
layout for the file?  It seems like the MDS needs to be able to use
SETATTR to the DS in order to implemenet fencing, so I wonder if the MDS
could perform such a SETATTR as part of issuing the layout.

> Assume that MDS.DOMAIN is a kdc that has a trust relationship with
> CORP.DOMAIN. Is it possible to use S4U to allow an authenticated
> [email-masked] on the NFS client to obtain a ticket for the
> principal "myfilelayout/[email-masked]" and set up a
> RPCSEC_GSS/krb5 session to a DS as that principal?

I think the logic for S4U ACL enforcement would be cleaner if it was using
the {host,nfs,root}/client.hostname credentials instead of (or in addition
to?) [email-masked].

In any case, my understanding is that the krb5 and S4U protocols have the
needed bits to permit that to happen, but actually implementing and
auditing the logic needed to do so in a secure fashion would be quite a
lot of work on the KDC for MDS.DOMAIN; I'm not sure that the current level
of granularity for configuring S4U acls is up for the task.

In a world where we are using flex files because we have storage
appliances that only speak NFSv3, it might be less work to replace the
appliances than to implement this S4U scheme.  So, it may be that
~"realistically, loose coupling is never going to be Kerberized." is the
most appropriate thing to say.

-Ben

---

From: Tom Haynes <[email-masked]>
Date: Mon, 2 Feb 2015 09:26:27 -0800

> On Feb 1, 2015, at 2:53 PM, Benjamin Kaduk <[email-masked]> wrote:
> 
> layout for the file?

Consider multiple users accessing the file.

>  It seems like the MDS needs to be able to use
>> [email-masked] <mailto:[email-masked]> on the NFS client to obtain a ticket for the
>> principal "myfilelayout/[email-masked] <mailto:myfilelayout/[email-masked]>" and set up a
> to?) [email-masked] <mailto:[email-masked]>.
> 
> appliances than to implement this S4U scheme.

But more $$$.

>  So, it may be that

Or, we could say that kerberos is more important than fencing and use the
owner of the metadata portion of the file as owner of the data portion.

---

From: David Noveck <[email-masked]>
Date: Mon, 2 Feb 2015 13:02:56 -0500

Ben said:

So, it may be that

I think your judgment as to future development is realistic, but I don't
think
that offering that judgment in a standards track document (which this is
going to have
to be)  is appropriate.

I think it does make sense to say that kerberizing loose coupling is
out-of-scope in the
context of the current document.  You would definitely still have the
option of a later
document defining a kerberizable loss-coupling mapping type.  It also might
be possible
to structure things to allow that to be defined later under the same
mapping type.

Tom said:

Right now, the ability to do fencing is a REQUIREMENT for a mapping type
according to
draft-ietf-nfsv4-layout-types-02.   I suppose that could change but I
hadn't anticipated
such drastic changes being made.

Still, if we are looking for material to assure a full agenda for Prague,
this might be an option :-).

>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>
>

---

From: Olga Kornievskaia <[email-masked]>
Date: Thu, 5 Feb 2015 13:06:16 -0500

Hi Tom,

Can you answer a couple more questions about flex layout spec?

1. In flexlayout, in GETDEVICEINFO operation, the client get's back
ff_device_addr4 that contains (a) a list of possible network address and
(b) a list of type ff_device_versions4. Is fs_device_version4 list in any
way prioritized? Is it interesting to have a notion of priority (say, nfsv3
is preferred)? Furthermore, each of ff_device_versions4 indicates
tightly/loosely. But what does it mean to get, for the same device, some
version that's tightly and another loosely? Or what does it mean to have in
one layout a combination of devices one being tightly-coupled and other
loosely-coupled?

2. I like the notion of statistics gathering and reporting in "lrf_boby" of
layout return_file4. However, I wonder what if after implementation you
find out that it would have been useful to include some other piece of
statistics that currently does not fit into the proposed data structures.
How easy would it be to add to flex layouts? Would it be useful to have a
switch on a type, like version1_stats, that describe current but allow for
version2_stats to include something else?

3. I'm still very confused about synthetic uids/gids for fencing. It seems
to be that it's not possible to fence just one client but you'll have to
fence all the client that had access to it. I'm not sure how uid/gids are
shared amount clients (ie., when multiple clients accessing the same file).
I see two options. For a given file, if you gave out the same synthetic
uid/gid to different clients. Then, fencing one fences them all.
Alternatively, if a file is owned but some random uid and gid. Then each
client that want access, are given individual uid and then their uid is
added to the gid of the file. Then again, if you need to revoke access,
you'll change gid of the file which would again deny access to everybody in
the group. I can see that alternatively, you can remove individual user
from the group, but in general "adding a user to the group or removing one"
can not be done via a SETATTR  so we are now talking about some kind of
"control protocol" for managing access and I thought loosely coupled stated
"no control protocol".

Thank you.

On Mon, Jan 5, 2015 at 6:14 PM, Spencer Shepler <[email-masked]>
wrote:

>
>
> Hi.  I am announcing the start of a Last Call for the Parallel NFS (pNFS)
> Flexible File Layout (draft-ietf-nfsv4-flex-files-04.txt) I-D.
>
>
>
> Please take time and review this document and provide feedback.  I believe
> this document is of particular interest to the WG and there has been
> productive discussion in the past on the content.
>
>
>
> The Last Call will end on January 20th.
>
>
>
> Spencer
>
>
>

---

From: Tom Haynes <[email-masked]>
Date: Thu, 5 Feb 2015 16:36:03 -0800

Hi Olga,

Thanks for the questions…

> On Feb 5, 2015, at 10:06 AM, Olga Kornievskaia <[email-masked]> wrote:
> 
> Hi Tom,
> 
> 1. In flexlayout, in GETDEVICEINFO operation, the client get's back ff_device_addr4 that contains (a) a list of possible network address and (b) a list of type ff_device_versions4. Is fs_device_version4 list in any way prioritized? Is it interesting to have a notion of priority (say, nfsv3 is preferred)?

No ordering.

Or in other words, you’d like the ffds_efficiency to be part of the device_id.  I think the choice of version for priority is a choice the client should make. 

> Furthermore, each of ff_device_versions4 indicates tightly/loosely. But what does it mean to get, for the same device, some version that's tightly and another loosely? Or what does it mean to have in one layout a combination of devices one being tightly-coupled and other loosely-coupled?

Hmm, it does look like this could have gone into ff_device_addr4 instead.

Except that while NFSv4 storage devices can be tightly-coupled, NFSv3 ones cannot.  I.e., there is no stateid to be passed.

As you point out below, for NFSv3 the fencing is against all clients. So, if the MDS tells the storage device to disallow all access, this could be seen as meeting the same requirements.

However, I think the expectation with tightly-coupled implementations is that only the “bad” client is impacted.

I’ll have to think more on this and expand on it in the document.

> 
> 2. I like the notion of statistics gathering and reporting in "lrf_boby" of layout return_file4. However, I wonder what if after implementation you find out that it would have been useful to include some other piece of statistics that currently does not fit into the proposed data structures. How easy would it be to add to flex layouts? Would it be useful to have a switch on a type, like version1_stats, that describe current but allow for version2_stats to include something else?

We’ve gone back and forth on this issue.

At the minimum you would need a process to get specifications for those versions in place.

We decided the overhead of defining new statistics in a new document was just as costly as creating a new version of the layout.

> 
> 3. I'm still very confused about synthetic uids/gids for fencing. It seems to be that it's not possible to fence just one client but you'll have to fence all the client that had access to it.

Yes that is true.

> I'm not sure how uid/gids are shared amount clients (ie., when multiple clients accessing the same file). I see two options. For a given file, if you gave out the same synthetic uid/gid to different clients. Then, fencing one fences them all. Alternatively, if a file is owned but some random uid and gid. Then each client that want access, are given individual uid and then their uid is added to the gid of the file. Then again, if you need to revoke access, you'll change gid of the file which would again deny access to everybody in the group. I can see that alternatively, you can remove individual user from the group, but in general "adding a user to the group or removing one" can not be done via a SETATTR  so we are now talking about some kind of "control protocol" for managing access and I thought loosely coupled stated "no control protocol”.

The group is for read only.

And the user is not part of that group - there is not a name service which maintains membership. The fact that the client presents the proper uid and gid determines access. I owe a new version of the document and I’ll give examples of the uid/gid interaction.

Look at Dave’s clarification on “control protocol”:

The coupling of the metadata server with the storage devices can
be either tight or loose.  In a tight coupling, there is a control protocol,
a purpose-built protocol (not specified here) present to manage security,
LAYOUTCOMMITs, etc.  In the case if loose coupling, the control protocol’s
functionality is more limited, and a version of NFS might serve the purpose.
Given the functionality available in such an arrangement, semantics
with regard to managing security, state, and locking models may be different
from those in the case of tight coupling and are described below.

I.e., NFS is not a purpose-built protocol but can act as a limited control protocol.

> 
> Thank you.
> 
> On Mon, Jan 5, 2015 at 6:14 PM, Spencer Shepler <[email-masked] <mailto:[email-masked]>> wrote:
>  
> 
> Hi.  I am announcing the start of a Last Call for the Parallel NFS (pNFS) Flexible File Layout (draft-ietf-nfsv4-flex-files-04.txt) I-D.
> 
>  
> 
> Please take time and review this document and provide feedback.  I believe this document is of particular interest to the WG and there has been productive discussion in the past on the content.
> 
>  
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/nfsv4 <https://www.ietf.org/mailman/listinfo/nfsv4>
> 
>

---

From: Olga Kornievskaia <[email-masked]>
Date: Fri, 6 Feb 2015 14:38:54 -0500

Thanks Tom,

On Thu, Feb 5, 2015 at 7:36 PM, Tom Haynes <[email-masked]>
wrote:

> is preferred)?
> Or in other words, you’d like the ffds_efficiency to be part of the
> device_id.  I think the choice of version for priority is a choice the
> client should make.
>
>
>
>
> Furthermore, each of ff_device_versions4 indicates tightly/loosely. But
> what does it mean to get, for the same device, some version that's tightly
> and another loosely? Or what does it mean to have in one layout a
> combination of devices one being tightly-coupled and other loosely-coupled?
>
>

So going beyond that. In the layout a client gets a set of mirrors. Each
minor has a deviceid and client does GETDEVICEINFO on that. So what does it
mean for one mirror to return a device that's tightly-coupled but another
mirror is loosely-coupled?

>
> Except that while NFSv4 storage devices can be tightly-coupled, NFSv3 ones
> cannot.  I.e., there is no stateid to be passed.
>
> As you point out below, for NFSv3 the fencing is against all clients. So,
> if the MDS tells the storage device to disallow all access, this could be
> seen as meeting the same requirements.
>

Is this really *just* NFSv3 and not loosely-coupled in general? For
loosely-coupled NFSv4 data servers, MDS would still be issuing synthetic
uid/gids and that's what's going to be used in fencing. In loosely-coupled
NFSv4 data servers no stateids are issued either, instead a zero-stateid is
used, correct?

> 2. I like the notion of statistics gathering and reporting in "lrf_boby"
> of layout return_file4. However, I wonder what if after implementation you
> We decided the overhead of defining new statistics in a new document was
> just as costly as creating a new version of the layout.
>
>
>
>
> fence all the client that had access to it.
>
>
> Yes that is true.
>
>
> I'm not sure how uid/gids are shared amount clients (ie., when multiple
> clients accessing the same file). I see two options. For a given file, if
> you gave out the same synthetic uid/gid to different clients. Then, fencing
> one fences them all. Alternatively, if a file is owned but some random uid
> and gid. Then each client that want access, are given individual uid and
> then their uid is added to the gid of the file. Then again, if you need to
> revoke access, you'll change gid of the file which would again deny access
> to everybody in the group. I can see that alternatively, you can remove
> individual user from the group, but in general "adding a user to the group
> or removing one" can not be done via a SETATTR  so we are now talking about
> some kind of "control protocol" for managing access and I thought loosely
> coupled stated "no control protocol”.
>
>
>
> And the user is not part of that group - there is not a name service which
> maintains membership. The fact that the client presents the proper uid and
> gid determines access. I owe a new version of the document and I’ll give
> examples of the uid/gid interaction.
>
>> Please take time and review this document and provide feedback.  I
>> believe this document is of particular interest to the WG and there has
>> been productive discussion in the past on the content.
>>
>>
>>

---

From: Tom Haynes <[email-masked]>
Date: Mon, 9 Feb 2015 10:30:33 -0800

> On Feb 6, 2015, at 11:38 AM, Olga Kornievskaia <[email-masked]> wrote:
> 
> Thanks Tom,
> 
> On Thu, Feb 5, 2015 at 7:36 PM, Tom Haynes <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Hi Olga,
> 
>> On Feb 5, 2015, at 10:06 AM, Olga Kornievskaia <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> Hi Tom,
>> 
> So going beyond that. In the layout a client gets a set of mirrors. Each minor has a deviceid and client does GETDEVICEINFO on that. So what does it mean for one mirror to return a device that's tightly-coupled but another mirror is loosely-coupled?

In general, the biggest point would be that the tightly-coupled mirror would handle fencing better.

Some side-effects

1) The tightly-coupled mirror could be able to exchange performance information with the MDS.
2) Kerberized access would be easier.
     a) The same principals are assigned to owner and owner_group
3) Calculating things like size can be easier.
4) Snapshots could be coordinated.

I.e., the control protocol allows the MDS and storage device to communicate or act on things outside the scope of NFS.

> Is this really *just* NFSv3 and not loosely-coupled in general? For loosely-coupled NFSv4 data servers, MDS would still be issuing synthetic uid/gids and that's what's going to be used in fencing. In loosely-coupled NFSv4 data servers no stateids are issued either, instead a zero-stateid is used, correct?

Given the set of operations allowed on the storage devices, yes.

I.e., if the set of operations allowed an OPEN with a FH, then we could get a valid stateid.

But as OPEN is not an allowed operation (or I guess we could say we should avoid it), then the Anonymous Stateid should be used. This is fine because there should not be an existing open on that file.