Subject: [nfsv4] FedFS Meeting Minutes, 03/19/2009

From: James Lentini <[email-masked]>
Date: Thu, 19 Mar 2009 15:58:42 -0400 (EDT)

FedFS Meeting Minutes, 03/19/2009
---------------------------------

Attendees
---------

Andy Adamson (NetApp)
Craig Everhart (NetApp)
David Gutierrez (?)
Renu Tewari (IBM)
Robert Thurlow (Sun)
Paul Lemahieu (EMC)
James Lentini (NetApp)

Minutes
-------

+ IETF Note Well Agreement

  This is a reminder that our discussions are governed by the 
  IETF Note Well Agreement. See:

    http://www.ietf.org/NOTEWELL.html

  We will start each week's meeting with this announcement.

+ Review Action Items

  [AI] Renu Tewari will investigate how rsync handles ACLs

  Skip (Renu had not joined yet).

+ NFSv4 WG Charter Update

  Hurray!

+ NFSv4 name mapping in a multi-realm environment

  Andy Adamson presented the slides he sent to the nfsv4 
  mailing list on 3/6/09.

  The work Andy presented was completed while he was a member of 
  UMich CITI in 2005. This work was funded by the TriLabs: LLNL,
  Sandia, and Los Alamos.

  The project's goal was to provide the means to replace 
  AFS with NFSv4. 

  AFS had some key advantages/attributes:
	- there was a one-to-one mapping between AFS cells and Kerberos realms
	- AFS had two useful group ACLs:
		- system any user: a user that does not have any credentials
		- system auth user: a user with kerberos credentials in the
		  AFS cell

  The world of NFSv4 is more complicated, since:
	- existing Kerberos configurations must be supported (AFS typically
	  introduced Kerberos to an environment, so requiring the one-to-one
	  mapping was not an issue)
	- NFSv4 does not mandate a particular on-disk format. AFS had an 
	  on disk format with its own UID/GID namespace

  The high level goal is to present a distributed /etc/passwd and /etc/group. 
  RFC2307 describes how to map name information into LDAP.

  The difficulty is that there may be multiple Kerberos realms within one 
  DNS domain, Kerberos realms that span DNS domains, etc. An NFSv4 Domain 
  may contain multiple DNS domains and/or Kerberos realms (see Andy's slides 
  for a picture).

  The proposed solution was to add two fields to the LDAP schema to accommodate 
  mapping in a multi-NFSv4 domain environment. 

  The slides contain many details. Some of the slides show multiple 
  client-side mappings in the Linux implementation. These have been removed by 
  improvements to the user/kernel interfaces (names passed using xattributes)
  since the presentation was created.

  This appears to be a problem relevant to NFSv4 in general. FedFS 
  may act as an accelerant.

  What aspects of this solution are potential IETF material?
  1. The LDAP schema changes
  2. Support for AFS-like auth any user and auth sys user ACLs

+ IETF NFSv4 Working Group Presentation

  NFSv4 WG meeting is March 25, 9:00 AM - 11:30 AM.

  We reviewed the slide deck prepared by James and Renu.

  The goal is to highlight key topics:
	- root fileset
	- LDAP and LDAP schema
	- security considerations
	- multi-NFSv4 domain UID/GID mapping

  We also briefly discussed a proposal that was raised at the FAST'09 
  FedFS BOF: To allow for other FSN resolution protocols, an FSN 
  could be composed of <NSDB name, FSN UUID, Resolution Protocol>.

  While this had seemed reasonable at the BOF, it was quickly pointed 
  out that this would lead to a many incompatible resolution 
  protocols. The agreement was that one protocol should be 
  sufficient.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 20 Mar 2009 09:01:21 -0400

>  protocols. The agreement was that one protocol should be 

I'm going to slightly disagree with this.

I agree that one protocol should be sufficient, as of now, and multiple
protocols now would lead to interoperability problems.

However, I do like the idea of having an FSN be composed of <NSDB name,
FSN UUID, Resolution Protocol>.  

The requirement in the protocol for now would be that the Resolution
Protocol would be required to be "FedFsV1" or another value designated
in a standards-track RFC which defines that new resolution protocol.

This would avoid incompatible protocols now but still leave the door
open for our successors to define a new protocol and convert to it
without extraordinary hassles.

-----Original Message-----
From: Lentini, James 
Sent: Thursday, March 19, 2009 3:59 PM
To: [email-masked]
Subject: [nfsv4] FedFS Meeting Minutes, 03/19/2009

Attendees
---------

Andy Adamson (NetApp)
Craig Everhart (NetApp)
David Gutierrez (?)
Renu Tewari (IBM)
Robert Thurlow (Sun)
Paul Lemahieu (EMC)
James Lentini (NetApp)

Minutes
-------

+ Review Action Items

  Hurray!

  AFS had some key advantages/attributes:
	- there was a one-to-one mapping between AFS cells and Kerberos
realms
	- AFS had two useful group ACLs:
		- system any user: a user that does not have any
credentials
		- system auth user: a user with kerberos credentials in
the
		  AFS cell

  The world of NFSv4 is more complicated, since:
	- existing Kerberos configurations must be supported (AFS
typically
	  introduced Kerberos to an environment, so requiring the
one-to-one
	  mapping was not an issue)
	- NFSv4 does not mandate a particular on-disk format. AFS had an

  The high level goal is to present a distributed /etc/passwd and
/etc/group. 
  RFC2307 describes how to map name information into LDAP.

  The difficulty is that there may be multiple Kerberos realms within
one 
  DNS domain, Kerberos realms that span DNS domains, etc. An NFSv4
Domain 
  may contain multiple DNS domains and/or Kerberos realms (see Andy's
slides 
  for a picture).

  The proposed solution was to add two fields to the LDAP schema to
accommodate 
  mapping in a multi-NFSv4 domain environment. 

  The slides contain many details. Some of the slides show multiple 
  client-side mappings in the Linux implementation. These have been
removed by 
  improvements to the user/kernel interfaces (names passed using
xattributes)
  since the presentation was created.

  While this had seemed reasonable at the BOF, it was quickly pointed 
  out that this would lead to a many incompatible resolution 
  protocols. The agreement was that one protocol should be 
  sufficient.
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: James Lentini <[email-masked]>
Date: Fri, 20 Mar 2009 10:26:36 -0400 (EDT)

> without extraordinary hassles.

The discussion on the call yesterday was about having alternative 
resolution protocols. For example, allowing some FSN's to be resolved 
with LDAP and others with an ONC RPC protocol.

Versioning the resolution protocol strike me as reasonable. My 
understanding is that LDAP can provide schema version information. If 
LDAP was the resolution protocol, the schema could be updated 
independently.

This brings up an interesting question about our confidence in 
mandating LDAP as the resolution protocol. We've entertained the idea 
of defining an ONC RPC resolution protocol for NFS server to NSDB 
communication. This would separate the NSDB protocol(s) from the 
NSDB's database format.

-james

---

From: Daniel Ellard <[email-masked]>
Date: Fri, 20 Mar 2009 07:29:58 -0700

> without extraordinary hassles.

Since we've moved to the nfsv4 mailing more recently than the last time
this came up for discussion, there are probably some readers who are
not familiar with all of the background discussion we had two years ago
or so.  I'm going to try to summarize that discussion so people can see
how we got where we are, and decide whether or not they like it.  This is
just background, not meant to agree or disagree with your point.

There was some initial discussion of having multiple resolution protocols,
but most of the other resolution protocols were degenerate: for example,
and FSN might look like <nfsv4, somehostname.com:/some/path?mount-options>:
pointing to a particular path on a host with a particular name forevermore.

The discussion of these forms of FSNs died away because in the
fedfs requirements discussion we realized that it was important that
FSNs be immutable.  If they're going to be embedded in multiple filesets,
owned by different admins, then it is completely impractical to update
an FSN when some attribute of an FSL changes.  For example, if we
wanted to move the fileset from somehostname.com to someotherhost.com,
we would need to find all of the FSNs that pointed to the original FSL and
change them.  Impossible--we don't know where they all are, nor are they
necessarily accessible (they could be offline, spun down, on archive media,
or admined by someone who simply doesn't trust us to touch their volumes).

In the long run, everything about the implementation of the FSLs that we
hard-code into the FSN is going to become wrong, and we'd like this to last
for a long time, so let's not hard-code anything.

My hunch is that hard-coding a particular resolution protocol reference into
an FSN is also wrong, because that could change as well, and if it does then
I don't want to have to tackle the impossible task of backpatching all of the
instances of that FSN (for the aforementioned reasons).  I think that the
protocol should be represented in the NSDB; let the NSDB tell you, the
referrer, how to find the referrant.

(Now of course someone is going to point out that encoding the NSDB name
in the FSN is a mistake for the reasons that I mentioned before--an NSDB
could change.  True, but we can manage this change, and it also greatly
reduces the number of things we need to control.  We can take steps to
ensure that there's an NSDB at nsdb.bbn.com, for example, much more
easily than that there's a fileset at foo.bbn.com:/some/path.)

The second point related to this discussion is another topic that has come
up several times, and should be mentioned here also.  The fedfs resolution
protocol is not *the* protocol, it is *a* protocol for FSN resolution.  We've
intentionally left plenty of wiggle room for implementors.  All we've done
is specify a protocol that must work (something any fedfs server can
count on) but not that all servers must use it.  Most of the underspecification
was intended to be for things like caching, hints, and the possibility of
consulting local NSDBs that proxy remote NSDBs instead of always
asking the NSDB named in the FSN, but it could also be used for things
like different protocols.

So, my hunch is that this isn't the right way to address this problem,
although I recognize the importance of the problem and I also recognize
that I'm not going to win any arguments without a more convincing
argument than a hunch!  An excellent topic for additional discussion.

-Dan

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 20 Mar 2009 15:42:03 -0400

> communication. This would separate the NSDB protocol(s) from the 

If the result of that discussion was anything other than "An RPC
resolution protocol would be totally stupid now and would remain stupid
forever.  I can't conceive of future working group members being so
idiotic as to go down that route." :-), then having the ability to later
specify that protocol in an RFC and identify its use via a Resolution
Protocol Name in the FSN seems worth providing, even if it turns out it
is never used.

-----Original Message-----
From: Lentini, James 
Sent: Friday, March 20, 2009 10:27 AM
To: Noveck, Dave
Cc: Lentini, James; [email-masked]
Subject: RE: [nfsv4] FedFS Meeting Minutes, 03/19/2009

-james

---

From: James Lentini <[email-masked]>
Date: Fri, 20 Mar 2009 16:07:00 -0400 (EDT)

On Fri, 20 Mar 2009, Noveck, Dave wrote:

> idiotic as to go down that route." :-), 

That wasn't the result. One reason to prefer this alternative is that 
by separating the resolution protocol from the database format allows 
for the database format to evolve independently of FedFS.

Allowing the database format to evolve over time is an important 
consideration. For example, the introduction of fs_locations_info in 
NFsv4.1 added new referral information beyond that found in 
fs_locations. The current NSDB database format contains all the 
information necessary to generate an fs_locations_info. If in the 
future, an fs_locations_info_improved was to be introduced with even 
more information, that would require an updated to the NSDB's schema. 
Clearly the resolution protocol would need to be modular enough to 
accommodate variable-sized responses.

> then having the ability to later specify that protocol in an RFC and 
> identify its use via a Resolution Protocol Name in the FSN seems 
> worth providing, even if it turns out it is never used.
>

---

From: "Sergey Klyushin" <[email-masked]>
Date: Mon, 23 Mar 2009 16:14:17 -0400

Hello,

Could you clarify the following?
>From Section 5.3. Named Attributes

   o  If OPENATTR is done on a named attribute directory or on a named
attribute, the server MUST return NFS4ERR_WRONG_TYPE.

>From Section 18.17. Operation 19: OPENATTR - Open Named Attribute
Directory:

If the current file handle designates an object of type NF4NAMEDATTR (a
named attribute) or NF4ATTRDIR (a named attribute directory), an error
of NFS4ERR_WRONG_TYPE is returned to the client.  

However the next sentence says: 

Name attributes or a named attribute directory may have their own named
attributes.

Should it be "MAY NOT" or what operation should be used to access named
attributes of named attribute/ named attribute directory?

Thanks in advance,
Sergey

---

From: "Mike Eisler" <[email-masked]>
Date: Wed, 25 Mar 2009 07:51:47 -0700 (PDT)

> attributes of named attribute/ named attribute directory?

OPENATTR of a named attribute directory doesn't make much sense to me.

OPENATTR of a named attribute might make sense.  I don't know. I vaguely
recall the discussions in the 1990s but am not sure what the reasoning was.
Regardless we will want to make the text consistent in the NFSv4.1 RFC.

What do implementations of named attributes do today?

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 25 Mar 2009 17:41:38 -0500

> Regardless we will want to make the text consistent in the NFSv4.1  

Initially, there was a desire to allow for this type of flexibility.   
However, in the
course of the 4.1 work that desire has decreased such that OPENATTR
on a named attribute is not that desirable.

Spencer

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 26 Mar 2009 09:33:38 -0400

We don't support OPENATTR of Named attributes or a named attribute
directory on Windows.
We treated "Name attributes or a named attribute directory may have
their own named attributes." text as inconsistency left over from
NFSv4.0.

BTW, there is one more typo: "Name attributes" instead of "Named
attributes"

Thanks,
Sergey

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Mike Eisler
Sent: Wednesday, March 25, 2009 10:52 AM
To: [email-masked]
Subject: Re: [nfsv4] Named attributes clarification

OPENATTR of a named attribute might make sense.  I don't know. I vaguely
recall the discussions in the 1990s but am not sure what the reasoning
was.
Regardless we will want to make the text consistent in the NFSv4.1 RFC.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 26 Mar 2009 10:18:26 -0400

I have not verified this by archaeological research on intermediate
drafts but as I recall, when the question was first raised about how
general a structure we might have under an OPENATTR, there were two
sorts of views:

     We should be restricting things to what I think 
     everybody implemented which is a single level of 
     directory with only the attribute equivalent of 
     regular files (created by open and with the semantics 
     of type NFREG but type returned as NF4NAMEDATTR).
     Nothing else with no named attributes for
     named attributes.

     "There are more things in heaven and earth...".  This
     is best left up to the file system that the server is 
     exporting to determine.

I believe that initially the second position was predominant and those
of the other persuasion weren't of a mind to have a big argument.

Later when we started to pay more attention to specific error detail,
the issue that the types structure was defined for something restricted
to the first environment and then the shoe was on the other foot.
People who held the second view would have to expand the type set a bit
for it to be both adequate and consistent and they didn't feel like
doing that.  For example,

     If you can do a CREATE in named attribute directory, then
     what's the type of the resulting object.  For example, if
     you create a directory is it NFDIR? It could be but then
     you have an asymmetry in what would be regular files
     (created by OPEN) in any other directory have a special type
     but what would be directories or symlinks don't.

     If you do an OPENATTR if a object of type NF4ATTRDIR the only
     type for the resulting object is an NF4ATTRDIR (and not some
     special type NF4ATTRDIRATTRDIR), so if you have the fh and
     get the type, you really don't know where you are.

So I think the specific statements you quote reflect what the group
decided when it focused on the WRONG_TYPE error.  I don't recall anyone
objecting.  The other statement:

     Name attributes or a named attribute directory may have 
     their own named attributes.

was inadvertently left in and reflected the second point of view which
the group had drifted away from.

For 4.1, I think we need to be clear which path we're taking here.  I
think we should restrict this to a simple model but we have to take
account of what people have implemented.  There's a lot of stuff that
most people (the sane ones, I guess) would never think of but is not
explicitly ruled out in RFC3530:

     A named attribute having its own named attributes. (Both
     ruled out and in).

     A sub-directory in the named attributes directory.

     A symlink in a named attributes directory.

     Other special file types in the named attribute 
     directory.

     The named attributes directory being in a different
     filesystem than the file for which it is a named
     attribute or similarly for named attribute subdirectories
     if they are allowed.

     LINK between named attributes.  The text says "files"
     which might exclude named attributes but in many
     cases "files" is used when named attributes are
     allowed.

     LINK between named attributes for different files.

     LINK between a file and a named attribute.  What 
     is the type of the object at that point?

We don't want to put in restrictions that interfere with existing
implementations and we don't want to put in new restrictions in 4.1 at
this stage, but I think that many of these are so "out-there" that the
lack of explicit prohibition is the sort of oversight that we might
correct at this stage.

-----Original Message-----
From: Sergey Klyushin [mailto:[email-masked]] 
Sent: Monday, March 23, 2009 4:14 PM
To: [email-masked]
Subject: [nfsv4] Named attributes clarification

Hello,

Directory:

Thanks in advance,
Sergey