Subject: [nfsv4] [FedFS] Meeting Minutes, 3/18/2010

From: James Lentini <[email-masked]>
Date: Thu, 18 Mar 2010 15:36:38 -0400 (EDT)

FedFS Meeting Minutes, 3/18/2010
--------------------------------

Attendees
---------

Craig Everhart (NetApp) 
Sorin Faibish (EMC)
Chuck Lever (Oracle)
Paul LeMahieu (EMC)
James Lentini (NetApp)
Trond Myklebust (NetApp)
Robert Thurlow (Oracle)
Nico Williams (Oracle)

Minutes
-------

+ IETF Note Well Agreement

  This is a reminder that our discussions are governed by the 
  IETF Note Well Agreement. See:

    http://www.ietf.org/NOTEWELL.html

  We will start each week's meeting with this announcement.

+ IETF'77

  No feedback on presentation.

+ DNS SRV Updates

  See mailing list thread:

  http://www.ietf.org/mail-archive/web/nfsv4/current/msg07778.html

  Rob Thurlow had a few comments he has sent Craig.

  Rob Thurlow wondered if the draft should raise the possibility of 
  a NFS server consulting the DNS SRV RR as this leads to a 
  discussion of how a DNS SRV RR is translated into a NFS referral.

  Craig said he was agnostic on this. The proposal mentions this 
  to point out the possibility of a NFS server performing the lookup. 
  The goal was not to prescribe a particular deployment option, but 
  rather describe the mechanism and allow implementations to choose 
  if the client or server would perform the resolution.

  Rob Thurlow suggested that it be clear that the client perform the 
  DNS SRV resolution. Craig noted that there is text "strongly 
  suggesting this". Craig said he would review the text in these 
  paragraphs to ensure the intention was clear.

  Nico offered to review the draft.

+ Admin Protocol: Junction Size

  Rob Thurlow recapped the limitations he's encountered with 
  regards to junction size.

  Nico mentioned that it is not uncommon for LDAP servers to 
  server multiple domains on a single port.

  The question is how the fileserver can locate the NSDB entries for 
  different domains.

  Nico's original suggestion was for the NSDB configuration to be 
  compatible with existing LDAP deployments. As a result, we 
  introduced the concept of the NCE to allow the NSDB entries to be 
  located at an arbitrary location in the DIT. However, the NCE 
  increases the size of a junction. If this size increase is 
  unacceptable, then Nico would not consider it a show stopper 
  to view the NSDB as a dedicated service.

  In the case that information for multiple domains are contained 
  in a single directory, the directory's root DSE will have 
  a naming context entry for each domain.

  How often is this configuration used in practice? Active Directory 
  uses it for the Global Catalog. Not sure if there are other uses.

  Nico suggested a way to use the DSE entry to include the NCE 
  information. The DSE could have an "NsdbContainerEntry" or similarly 
  named entry that contains the base DN of the NSDB data.

  Rob suggested that it would be convenient to query on a domain 
  and then determine the NCE.

  Chuck asked about the persistent of junction objects. If the 
  domain is one of the items stored in a junction, is there a 
  requirement that the mapping be unchanged for the life of the 
  object?

  Rob is not concerned with this. It is OK if the mapping changes. 
  He is concerned with copying/moving/backing up a junction. He 
  wants to ensure that if a junction is copied (either as an 
  individual object or as part of the replication of an entire 
  fileset) that the junction remains consistent. He does not want 
  copy both the junction and separate data on which its interpretation 
  depends.

  Nico asked that copying a junction be defined. In this context,
  a junction copy is either performed via the Admin protocol or via 
  an implementation specific protocol.

  James asked if existing LDAP implementations allow for new root 
  DSE attributes such as the "NsdbContainerEntry" attribute described 
  above. The belief is that they do.

  Would there be one "NsdbContainerEntry" DSE entry per directory?

  Yes, this is the idea. 

  One option is to have it be a full DN that points to a single root 
  for all NSDB data stored in the directory. If there were multiple 
  domains in the directory, all there data would be located in here. 
  This will work because FSN UUIDs and FSL UUIDs are unique across domains.

  Another option is for the contents of the "NsdbContainerEntry" to 
  be a string that is pre-pended to each NamingContext in the DSE 
  to obtain a location for the NSDB entries. This would require the 
  NSDB container to be in the same relative location for all domains 
  hosted by the NSDB.

  Would DNS be an appropriate place to map from a domain name to a 
  NCE value?

  Nico's impression is that the DNS community would prefer DNS not 
  be used for this type of application. Nico likes the idea of using 
  LDAP since the NCE is an LDAP DN.

  Nico noted that Active Directory has its own SRV RR discovery service. 
  Nico would like us to copy this. We've discussed the standardization 
  of this before. A previous attempt to standardize a generic LDAP SRV 
  RR format failed. Nico believes that a NSDB specific format would 
  be acceptable since it is for a specific purpose.

  A document describing the AD SRV RR format is here:
  http://technet.microsoft.com/en-us/library/cc961719.aspx

  James pointed out that locating the NSDB and locating the NCE are 
  different subjects.

  Nico will research the DSE entry option further with experts in the 
  LDAP community.

+ Admin Protocol: feedback from Chuck Lever

   - FedFsUuid XDR Type

     A FedFsUuid is defined as a variable size opaque array with a 
     maximum of 16 octets (XDR encoded this as a 4 octet length 
     followed by up to 16 octets; this requires 20 octets for an 
     RFC 4122 UUID):

       typedef opaque FedFsUuid<16>;

     since RFC 4122 UUIDs are always 16 octets, a fixed length 
     array would be a more efficient (encoded as 16 octets):

       typedef opaque FedFsUuid[16];

     We agreed that this should be included in the next version of 
     the draft.

   - Optional LDAP error value

     Rather than make the LDAP error value optional for 
     FEDFS_ERR_NSDB_LDAP, two FEDFS_ERR values could be defined:    
     one to indicate that an LDAP error occurred and one to indicate
     an LDAP error occurred and that the LDAP error code is available.
     See: 
     
     http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admin-rfcdiff.html

     There were not strong opinions on this. 
     We will include it in the next version of the draft.

   - International DNS names

     How are International DNS names encoded in a FedFsNsdbName?

     One option: handle UTF-8 strings as defined by the 
     nfs4_cs_prep and nfs4_cis_prep profiles in Chapter 14 of 
     RFC 5661. The nfs4_cs_prep profile (cs == case sensitive) 
     would be used for pathnames, and nfs4_cis_prep profile 
     (cis == case insensitive) would be used for DNS names. 
     We'll need some time to review this approach before we can 
     decide if it is right for FedFS.

     Rob Thurlow thought that it would be reasonable for most 
     implementers to follow the precedent set by NFS.

     Nico would like to store the Punycode strings and use Punycode 
     strings on the wire. He noted that UTF-8 should be used 
     for displaying the string in UIs.

     Chuck thought that we should keep with the NFSv4 precedent.

     Chuck: What will be stored in the NCE string?
     James: This will be an LDAP DN. 

     Nico pointed out that this will be used in both the Admin protocol 
     and the LDAP protocol. Need the encoding to work for both.

     James pointed out that the discussion on junction size 
     will determine if the NCE is part of the Admin protocol.

     Nico noted that our approach should be consistent with 
     RFC 3490 and RFC 3491. Chuck noted that RFC 4690 was 
     also relevant.

     We agreed that the main thing was to be consistent.

   - Pathnames

     Q. If the pathname in FEDFS_JUNCTION_CREATE is not relative 
     to the NFS pseudo-root, how does the Admin client generate 
     it?

     A. As implied by the text in the second paragraph of Section 
     5.1 (page 12), pathnames in the Admin protocol are 
     implementation dependent.

     Implementation dependent pathnames are used for the 
     following reasons:

     * The Admin protocol is designed to be agnostic of any 
       particular file access protocol. The namespace managed 
       by the Admin protocol may allow access by NFS, CIFS, etc. 
       clients.

     * A unique external path may not exist since:

       - An object's path depends on the NFS protocol version 
         (e.g. the NFSv3 path is not the same as the NFSv4 path).

       - Servers might show different pseudo-fs's to different 
         clients.

    * Allows junctions to be created before their 
      containing fileset is exported.

    Nico felt that this made the administration task hard on a user.

    Chuck would like there to be a way for paths to be discovered.

    Rob would like to have paths relative to the pseudo-root.

    Chuck pointed out that the FedFS admin server might be running 
    in a jail or have its own local file system namespace (chroot). 

    Nico observed that each jail should have its own pseudo-root. 

    [We ran out of time on this topic. We will continue discussing 
     it at the next meeting]

+ Next Meeting

  No meeting next week. We will be attending the NFSv4 WG meeting.
  The next meeting will be April 1.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 18 Mar 2010 15:07:10 -0500

>   Nico offered to review the draft.

The DNS aspects of this I-D are OK, but I agree with Rob that it would
be preferable to punt all resolution issues to the client.  I just don't
see how we do that without the NFSv4.x protocol itself having a way to
return junction data to the client -- fs_locations_info4 and
fs_location4 effectively require resolving junctions on the server-side.

>   named entry that contains the base DN of the NSDB data.

Right.  There's at most two things to indicate in the root DSE (note:
it's "root DSE", not just "DSE"):

 - a DN, if there's a single container for all NSDB objects in this
   server;

 - an RDN, if there's a per-domain container for all NSDB objects for
   that one domain.

Unless we expect to have a situation such as:

    Server name                 Naming contexts served
    -----------                 ----------------------

    bar.foo.example             DC=a,DC=foo,DC=example and
                                DC=b,DC=foo,DC=example

    baz.foo.example             DC=a,DC=foo,DC=example

then the first option is good enough because NSDB objects are named
after UUIDs.  To be sure, the second option would be the safest.

The root DSE attribute name can be nsdbContainerRDN and its syntax could
be... well, there's no RDN syntax, only a DN syntax (see RFC4517; the DN
syntax is described in section 3.3.9), but I suspect that DN syntax will
do.

>        by the Admin protocol may allow access by NFS, CIFS, etc. 

If so we should actually do the work of showing how to use the ADMIN
protocol in connection with two sufficiently different file access
protocols.

IMO for now pathnames used in the ADMIN protocol should always be
relative to the NFS pseudo-root.  When we add ADMIN protocol support for
other protocols we can add options to deal with any protocols that lack
a notion of pseudo-root and cannot be mapped onto a pseudo-root.

>          (e.g. the NFSv3 path is not the same as the NFSv4 path).

Are there any NFSv3 and v4 implementations where that is actually the
case?

>        - Servers might show different pseudo-fs's to different 

The ADMIN server should have the same exact view of the filesystem
namespace as the file server that it is used to ADMIN.

>       containing fileset is exported.

This is _almost_ indistinguishable from the previous item.

>     Nico felt that this made the administration task hard on a user.

But I don't think it's fatal, just annoying.

>     Nico observed that each jail should have its own pseudo-root. 

And that chrooted ADMIN servers should know the chroot path and thus be
able to strip that prefix from pathnames sent by ADMIN clients.

>     [We ran out of time on this topic. We will continue discussing 

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Thu, 18 Mar 2010 17:03:56 -0400

> protocol in connection with two sufficiently different file access

I agree, real world use cases would be helpful.

> Are there any NFSv3 and v4 implementations where that is actually the

I think there are, but the NFSv4 servers in that case are already 
considered to be broken in this way.  (RHEL 5, or older releases of 
NetApp's OnTAP 6, for example).  The solution may be to agree not to 
support such implementations.

> namespace as the file server that it is used to ADMIN.

That may contradict some statements in the admin protocol spec draft; 
see Chapter 5 section 1.

> But I don't think it's fatal, just annoying.

For the record, my main concern is that the admin protocol should not be 
so general that it's difficult or impossible for a FedFS admin client 
implementer or an end-user admin to construct junction pathnames.  A 
little "annoying" can go a long way when faced with the task of 
administrating hundreds of file servers.  I assume a lot of this 
complexity will be hidden by the admin client implementation; how much 
can be hidden is up to the admin protocol design.

It would be nice, for instance, to provide a way for an admin to browse 
the server's file system to find the right place to create a junction. 
(Can I do this by probing with FEDFS_LOOKUP_FSN?)

I wouldn't expect that every real world file server would have a 
constant and well-known layout so that remote admins would know exactly 
where to put junctions without looking.  The spec implies that a remote 
admin can't rely on the server's file system even being exported.

To manage hundreds of file servers, we might also expect there to be 
some form of automation in the admin client.

-- 
chuck[dot]lever[at]oracle[dot]com

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 18 Mar 2010 17:20:04 -0500

> I agree, real world use cases would be helpful.

I'll come back to this below.

> support such implementations.

But all of those have NFSv4 support, and can share all the same
filesystems with NFSv4 even when also using NFSv3, correct?  If so I
think we should just ignore NFSv3 and focus of NFSv4 and, possibly,
CIFS.

> That may contradict some statements in the admin protocol spec
> draft; see Chapter 5 section 1.

It is a *draft*, we can change it yet.  And I agree with the
parenthetical in that paragraph: having the paths to the same directory
differ when via the file access protocol versus the ADMIN protocol is
just plainly unhelpful, if not outright useless.  (I'm generalizing; the
text in question specifically mentions NFS, but the target directory
could be accessible via CIFS.)

> For the record, my main concern is that the admin protocol should
> not be so general that it's difficult or impossible for a FedFS
> admin client implementer or an end-user admin to construct junction
> pathnames.  A little "annoying" can go a long way when faced with
> the task of administrating hundreds of file servers.  I assume a lot
> of this complexity will be hidden by the admin client
> implementation; how much can be hidden is up to the admin protocol
> design.

This is true, and, for me, convincing.

> It would be nice, for instance, to provide a way for an admin to
> browse the server's file system to find the right place to create a
> junction. (Can I do this by probing with FEDFS_LOOKUP_FSN?)

This brings up another issue, section 5.1 says:

   The last component of the path MUST be an empty directory.

and since the ADMIN protocol can't be used to create that directory,
there's no way to create the directory and the junction atomically.  But
then, NFSv4 doesn't have a way to atomically combine operations in
COMPOUND anyways, so I suppose there's no harm in this.

(Also, traditionally one could mount files, not just directories, but if
the junction has to be on a directory...  Whatever.)

> constant and well-known layout so that remote admins would know
> exactly where to put junctions without looking.  The spec implies
> that a remote admin can't rely on the server's file system even
> being exported.
> 

Here's my suggestion:

a) Define ADMIN pathnames relative to a server's NFSv4 pseudo-root.

b) Leave CIFS and other filesystems for another day, making sure that
   the ADMIN protocol is sufficiently extensible that we can actually do
   that.

c) Remove the possibility of a pathname being accessible via ADMIN but
   not the file access protocol.  The path need not be accessible via
   the file access protocol for all clients, but it must be for the
   ADMIN client.

   If the ADMIN client is using a CIFS path then that path must be
   accessible via CIFS.  Same if via NFSv4.

This should greatly simplify the ADMIN protocol, and implementations.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Fri, 19 Mar 2010 10:14:01 -0400

> From: Chuck Lever [mailto:[email-masked]] on March 18, 2010
5:04 PM 
> I agree, real world use cases would be helpful.

Hear, hear.  I co-wrote this spec and my eyes still cross when I try to
follow some of the namespace arguments.  There are so many choices for
namespaces here and we don't have good names for them in arguments like
these.  Every variation on protocol or administrative access seems to be
able to introduce variations in that namespace.

> It would be nice, for instance, to provide a way for an admin to
browse
> (Can I do this by probing with FEDFS_LOOKUP_FSN?)

I think the right place to do this is in the base file server protocol,
NFSv4 in this case.  That's explicitly out of the charter of the FedFS
work, as I understand it, so such an effort couldn't really result from
the FedFS call group.  But choosing an origin/source for a junction by
navigating there feels like a far less contentious method, and requires
the least amount of protocol copying.

If an admin wants to build a referral tree before exporting it, that
case could almost be supported by using ACLs to restrict access
administratively to the exported tree-in-progress, building it, then
un-restricting access.

		Craig

---

From: "Everhart, Craig" <[email-masked]>
Date: Fri, 19 Mar 2010 10:33:27 -0400

Inline.

> From: Nicolas Williams [mailto:[email-masked]]
> Sent: Thursday, March 18, 2010 4:07 PM
> 

I may not have expressed the idea clearly enough.  But if you really
think that SRV data isn't enough for a server to construct fs_location*
information, then SRV perhaps isn't enough for clients to follow the
name pointer either.

Yes, the idea is that the server "resolv[e] junctions" on the
server-side, but what I mean by that is that they'd do the DNS lookup
and massage the result into objects with fs_location* attributes.

The idea, perhaps not expressed clearly enough, is that some local
installation could create a local NFSv4 server with no real data of its
own, but which exported what amounts to /nfs4 in the projected
distributed name space.  An NFSv4 client would mount the magic directory
and do lookups and readdirs in it.  Every lookup would basically do the
SRV lookup.  The local NFSv4 server invents dummy directories under its
/nfs4 directory, but the only attributes that this server is willing to
return to its NFSv4 clients are from the fs_location* set of attributes.

The use case is that it allows an installation to support the wide-area
name space discovery without requiring every client to be updated to
read SRV records.

		Craig

---

From: Chuck Lever <[email-masked]>
Date: Fri, 19 Mar 2010 12:29:03 -0400

> think we should just ignore NFSv3 and focus of NFSv4 and, possibly,

We have several file server implementations that can support multiple 
file access protocols (Linux, Solaris, OnTAP).  I think that means CIFS 
must be considered _before_ finalizing the draft, even if the final spec 
doesn't explicitly support CIFS.

> could be accessible via CIFS.)

I would expect these paths to be different.  I'm not especially familiar 
with CIFS, but there is usually a share name at the start of exported 
CIFS paths, isn't there?

In addition, the NFSv4 pseudofs doesn't always precisely align with the 
server's real local file system.  In some cases, I think the pseudofs 
starts at the real / on the server, but leaves out any parts of the 
local file systems that aren't exported.

Some of the wording in the FedFS admin draft suggests it MUST still be 
possible to create a junction on the server's local file system over a 
directory that the admin has no intention of ever exporting (like, say, 
an empty directory in /etc).  Did I read that incorrectly?  That would 
be one possible reason to insist on running the FedFS admin service 
daemon in a chroot jail.

> the junction has to be on a directory...  Whatever.)

The pre-existing directory requirement is another scalability problem, 
IMO.  This guarantees that a FedFS admin creating a FedFS domain from 
scratch must either login to or mount every server filesystem on which a 
junction should be created, and pre-create these directories.  (That 
would add at least one more key/password onto the list of keys needed to 
perform FedFS domain administration, and possibly more than one).

IOW the FedFS admin protocol and the NSDB by themselves are not 
sufficient to construct a multi-server namespace.  A file access 
protocol (including ssh here) is also required.

In some ways, we might be better off managing junctions via file access 
protocols instead of a separate admin protocol.  The client tool would 
be used on a specific directory in a mounted file system, handling NSDB 
metadata checking itself before constructing the junction.

That would require prescribing the structure and content of the object 
that is a junction, for all file systems, however.

---

From: James Lentini <[email-masked]>
Date: Fri, 19 Mar 2010 17:43:29 -0400 (EDT)

> On Thu, Mar 18, 2010 at 03:36:38PM -0400, James Lentini wrote:
<snip>
>     Server name                 Naming contexts served

Since the FSN and FSL objects are named by UUIDs, there shouldn't be 
namespace collisions in the first option.

One complication with the second option is that the fileserver needs 
to search the NSDB container entry in every namingContext hosted by an 
NSDB (directory). A search of all namingContexts is required because 
the fileserver does not know a priori which one corresponds to the 
domain it is interested in. For example, d a domain named bar.example 
might map to the namingContext "O=bar.example" or some other variant, 
rather than "DC=bar,DC=example". My expectation is that the vast 
majority of NSDBs wil host the records for a single domain, so this 
search will be the exception, not the rule.

> syntax is described in section 3.3.9), but I suspect that DN syntax will

I was unable to create an entry in the root DSE of my OpenLDAP-based 
NSDB. Does anyone have instructions on how to make this work?

I used Apache Directory Studio (version 1.5.2.v20091211) to connect to 
my OpenLDAP directory (version 2.4.12). When I attempt to add a 
distinguishedName attribute to the root DSE, I received LDAP error 
code 53 and the following message "modify upon the root DSE not 
supported".

Based on our discussions yesterday, my understanding was that existing 
LDAP implementations would support modifying the root DSE.
 
> >    - Pathnames
<snip>
> Are there any NFSv3 and v4 implementations where that is actually the

Yes, Linux can be configured this way.
 
I used this as an example to illustrate that even within a given 
protocol, pathnames can differ across versions. I wasn't suggesting 
that support for NFSv3 pathnames was a requirement.

-james

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 19 Mar 2010 18:43:02 -0500

On Fri, Mar 19, 2010 at 12:29:03PM -0400, Chuck Lever wrote:
> We have several file server implementations that can support
> multiple file access protocols (Linux, Solaris, OnTAP).  I think
> that means CIFS must be considered _before_ finalizing the draft,
> even if the final spec doesn't explicitly support CIFS.

I'd say we need to consider and support extensibility, but must handle
at least NFSv4 correctly to start.  I don't think there's a need to
support CIFS from the start, but it's crucial that we have a good idea
of how we'll slot CIFS in later.  IOW, we agree violently :)

> > [...]
> 
> I would expect these paths to be different.  I'm not especially
> familiar with CIFS, but there is usually a share name at the start
> of exported CIFS paths, isn't there?

Yes.

> In addition, the NFSv4 pseudofs doesn't always precisely align with
> the server's real local file system.  In some cases, I think the
> pseudofs starts at the real / on the server, but leaves out any
> parts of the local file systems that aren't exported.

Irrelevant: the NFSv4 paths used in the ADMIN protocol should be exactly
those used in the NFSv4 protocol.  All other local paths on the server
should be ignored.

> Some of the wording in the FedFS admin draft suggests it MUST still
> be possible to create a junction on the server's local file system
> over a directory that the admin has no intention of ever exporting
> (like, say, an empty directory in /etc).  Did I read that
> incorrectly?  That would be one possible reason to insist on running
> the FedFS admin service daemon in a chroot jail.

That was my impression, but I think that's a bad idea.  See my other
comments re: chrooting.

> >[...]
> The pre-existing directory requirement is another scalability
> problem, IMO.  [...]
> 
> [...]
> 
> In some ways, we might be better off managing junctions via file
> access protocols instead of a separate admin protocol.  The client
> tool would be used on a specific directory in a mounted file system,
> handling NSDB metadata checking itself before constructing the
> junction.

I strongly agree, however I think this would be a non-starter at this
point.  Having the ADMIN protocol be able to create the required empty
directory is also problematic, since then one has to worry about ACLs
and other such attributes and we begin to replicate NFSv4 in the ADMIN
protocol -- another non-starter.

I think we can deal with this one requirement.  What strikes me as
unacceptable is the requirement that the ADMIN protocol be able to deal
with pathnames that are not yet shared/reachable via any file access
protocol.  The combination of those two requirements is too problematic.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 19 Mar 2010 18:48:14 -0500

On Fri, Mar 19, 2010 at 10:33:27AM -0400, Everhart, Craig wrote:
> > The DNS aspects of this I-D are OK, but I agree with Rob that it
> > would be preferable to punt all resolution issues to the client.  I
> > just don't see how we do that without the NFSv4.x protocol itself
> > having a way to return junction data to the client --
> > fs_locations_info4 and fs_location4 effectively require resolving
> > junctions on the server- side.
> 

I'm not saying that SRV RRs are insufficient (they are sufficient).  Rob
seems to be complaining that the server should not be the one doing the
SRV RR lookups, and I very much agree.  However, I don't see how to
avoid the server doing the lookups without revising NFSv4 since
fs_locations_info4 and fs_location4 deal in server names, not names-to-
be-resolved-by-SRV-RRs.

> and massage the result into objects with fs_location* attributes.

Yes, but that means that the server has to do long-running lookups
synchronously.  This presents problems for threaded-model
implementations of NFSv4 (e.g., ours) (this is not insurmountable of
course, just bloody annoying).

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 19 Mar 2010 18:54:32 -0500

> namespace collisions in the first option.

Right.

> domain it is interested in. [...]

Oh, well, I really think we need a standard mapping of DNS domainname to
LDAP base DN.  I realize that we don't have one.  MSFT implemented the
most obvious such mapping and that is now almost a de facto standard,
though plenty of "legacy" remains.  I reluctantly concede the point.

> NSDB. Does anyone have instructions on how to make this work?

I believe you need to install a schema that adds an auxiliary class to
OpenLDAProotDSE.

> that support for NFSv3 pathnames was a requirement.

I consider NFSv3 and NFSv4 to be radically different protocols that
happen to share a name.  They can barely be said to be different
versions of the same protocol (oh, sure, they run over the same port
number, so what).

---

From: Chuck Lever <[email-masked]>
Date: Sat, 20 Mar 2010 12:20:59 -0400

On Mar 19, 2010, at 7:43 PM, Nicolas Williams wrote:
> point.

Understood.

--
Chuck Lever

---

From: "Everhart, Craig" <[email-masked]>
Date: Sun, 21 Mar 2010 13:11:48 -0400

> Sent: Friday, March 19, 2010 7:48 PM

Eh?

I don't know what you're arguing.  I still believe that it's possible to
do this stuff on the client side or on the server side of a site's NFS
implementation.  That would be an architectural choice for whoever is
architecting, implementing, and installing that implementation.

Alternative A says that it's the NFSv4 client's job to do DNS lookups
and interpret SRV data.  So an NFSv4 client establishes a "magic
automounter", but still local, directory "/nfs4".  If an application
does a lookup for the previously-unknown name "/nfs4/example.net", that
NFSv4 client does a lookup for a SRV record
"_nfs4._domainroot._tcp.example.net".  It mounts the filesystem
"/.domainroot-example.net" on any of the file servers named, and it goes
from there.

Alternative B says that the NFSv4 client isn't involved with SRV data.
Maybe it's running an old client or one that can't be updated.  In any
case, some site decides that it cannot or does not want its clients
doing the DNS work.  They can in principle craft an NFSv4 server that
exports a directory with the same functionality as the conventional
"/nfs4" directory, and use it to handle SRV functionality for their
site.  The local NFSv4 client simply mounts that directory without any
expectations about global names.  When an application on the NFSv4
client does the lookup for "/nfs4/example.net", the special NFSv4 server
does the DNS lookup as above, massages the results into fs_location*
information, and returns the result.  As far as I understand it, the
results of the SRV query (the Target names in the RDATA sections) are
indeed server names for the correct NFSv4 servers, and they can be
returned in the fs_location* attributes.

Am I missing some complexity?

> course, just bloody annoying).

I'm not arguing that every server implementor needs to allow
administrators to build an "Alternative B" deployment.  Is that the
concern?  Would you be happier if this draft started using MUST/SHOULD
language here?  I'd be happy with

- MUST use one of the alternatives A or B
- SHOULD use alternative A
- MAY use alternative B

Is that what we're discussing?

		Craig

---

From: James Lentini <[email-masked]>
Date: Sun, 21 Mar 2010 15:59:51 -0400 (EDT)

> I believe you need to install a schema that adds an auxiliary class to

Thanks. Searching the OpenLDAP documentation for this led me to the 
solution. 

If anyone else is interested, here is how to get this working on 
OpenLDAP 2.4.12. I am interested in hearing what the experience is 
like with other LDAP implementations.

First, add a rootDSE directive to the slapd.conf file. Then create an 
LDIF file with the root DSE attributes to be added. Make sure that 
this file is owned (or at least readable) by the ldap user. 
Finally, restart slapd with the new configuration. My exact 
configuration is:

# grep rootDSE /etc/openldap/slapd.conf
rootDSE /etc/openldap/rootDSE.ldif
# ls -l /etc/openldap/rootDSE.ldif
-r-------- 1 ldap ldap 112 2010-03-21 14:58 /etc/openldap/rootDSE.ldif
# cat /etc/openldap/rootDSE.ldif
dn:
fedfsNsdbContainerEntry: o=fedfs,dc=foo,dc=example
fedfsNsdbContainerEntry: o=nsdb,dc=baz,dc=bar,dc=example
# /etc/init.d/ldap restart
Stopping slapd:                                            [  OK  ]
Starting slapd:                                            [  OK  ]
# ldapsearch -LLL -x -b '' -s base '(objectclass=*)' fedfsNsdbContainerEntry
dn:
fedfsNsdbContainerEntry: o=fedfs,dc=foo,dc=example
fedfsNsdbContainerEntry: o=nsdb,dc=baz,dc=bar,dc=example

> versions of the same protocol (oh, sure, they run over the same port

My understanding is that this observation (pathnames can be different 
between versions of the same protocol) is one of three 
goals/observations that led to the current approach. I think you are 
agreeing with this particular observation, although perhaps you 
consider it an extreme case of pathnames differing between protocol 
versions.

I think reviewing the original goals/observations is useful since many 
of us, including myself, were not present when the current approach 
was chosen.

-james

---

From: Nicolas Williams <[email-masked]>
Date: Sun, 21 Mar 2010 15:57:21 -0500

> Thanks. Searching the OpenLDAP documentation for this led me to the 

Excellent.  Thanks for confirming this.

> consider it an extreme case of pathnames differing between protocol 

In the case of NFS I think it's certainly possible to have exactly the
same pathnames (ignoring I18N issues / assuming UTF-8 and normalization-
insensitive file naming) for v3 and v4.  That some implementations do
something different is not terribly interesting to me because in those
cases one could still map a v3 path to a v4 path.

It's CIFS that is interesting.  Again, IMO the solution is:

 - The ADMIN protocol should have file access protocol specific pathname
   forms.  Whether this is a string and a file access protocol type ID
   or a union of protocol-specific pathname data types discriminated by
   file access protocol type is a less important detail.

 - The ADMIN protocol MUST, for each file access protocol type, have
   exactly the same file namespace as the file server located at the
   same IP address.

> of us, including myself, were not present when the current approach 

There's certainly still time to make changes to the protocol, no?  Or
has it been deployed already?

Nico
--

---

From: James Lentini <[email-masked]>
Date: Sun, 21 Mar 2010 17:59:05 -0400 (EDT)

>    file access protocol type is a less important detail.

I discussed this option with Dan Ellard last week. This option could 
allow for implementation specific paths to objects inaccessible via 
any file access protocol (see the FEDFS_ADMIN_PATH type below). This 
is something the original approach provided for.

typedef opaque FedFsPathComponent<>;
typedef opaque FedFsPathName<>;

enum FedFsPathType {
      FEDFS_ADMIN_PATH = 0,   /* an admin interface path */
      FEDFS_NFS4_PATH = 1     /* an NFSv4.* path         */
      /* more TBD (e.g. SMB) */
};

struct FedFsPath {
      FedFsPathType    type;
      FedFsPathName    path;
}

>    exactly the same file namespace as the file server located at the

That sounds about right to me, although I would suggest two additional 
conditions.

First, the file namespace is determined by the (client, server) pair 
to allow for servers capable of projecting a client-specific 
namespace.

Second, I would suggest a rule for the unusual case of multiple file 
servers at the same IP address.

For NFS, the rule might be that a pathname is for the file server 
identified as the NFSv4 server by RPCBIND.

> has it been deployed already?

This isn't set in stone. I believe a change like the one above would 
be relatively minor.

---

From: "Everhart, Craig" <[email-masked]>
Date: Sun, 21 Mar 2010 18:21:20 -0400

> From: Lentini, James
> Sent: Sunday, March 21, 2010 5:59 PM
> ...
....
> This isn't set in stone. I believe a change like the one above would

I think we could go a long way with a subset rule: any name usable on
the file server MUST be acceptable to the ADMIN protocol.  I think we'll
box ourselves in if that's the *only* pathname that is usable.

I'm not sure I yet understand the use cases for client-specific names.
I think I'd allow for them as other alternatives for naming objects on
the server, just not ones that are covered in the specification.

So I'd suggest the rule

  For any file access protocol type, any path usable on the file server
MUST be usable by the ADMIN protocol.

		Craig

---

From: Nicolas Williams <[email-masked]>
Date: Sun, 21 Mar 2010 18:56:05 -0500

On Sun, Mar 21, 2010 at 06:21:20PM -0400, Everhart, Craig wrote:
> > > There's certainly still time to make changes to the protocol, no?
> > > Or has it been deployed already?
> > 

I disagree.  The problem is that allowing the ADMIN protocol to refer to
objects not reachable through the file access protocols means requiring
local access to the file server (in order to create those objects).
This is not helpful for interop.  We can always extend the ADMIN
protocol later if we find that we did need such a thing after all.

Nico
--

---

From: Daniel Ellard <[email-masked]>
Date: Sun, 21 Mar 2010 20:41:13 -0400

> protocol later if we find that we did need such a thing after all.

This means that all of the elements of the namespace that the admin
is trying to create must be visible (exported) before the admin can do
anything.  It seems backwards to make the admin mount filesets in
their namespace in order to define where in their namespace the
fileset should appear...

A close analogy to setting up junctions in a fileset is setting
up a pseudo-fs.  Does anyone consider doing that remotely?

-Dan

---

From: Nicolas Williams <[email-masked]>
Date: Sun, 21 Mar 2010 22:09:02 -0500

> their namespace in order to define where in their namespace the

The ADMIN doc already demands that junctions can only be created on
empty directories and provides no mechanism by which such directories
can be created.  You have these options then:

 - Junction locations are created via local means (e.g., ssh in, type "mkdir
   <path>").

   This would be NOT interoperable because it'd be outside the spec.

 - Junction locations are created via file access protocols.

   But this restates the requirement we've been discussing.

 - Extend the ADMIN protocol to allow creation of junction locations
   directly via the ADMIN protocol.

   But this is complicated since: a) it replicates functionality from
   the file access protocols, b) it requires support for ACLs and other
   such attributes, which only makes (a) worse.

> up a pseudo-fs.  Does anyone consider doing that remotely?

That's the whole point of the ADMIN protocol!

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 22 Mar 2010 12:56:21 -0400

>     exactly the same file namespace as the file server located at the

I'm having trouble wrapping my head around this.

Suppose we have a file server that exports a given local file system via 
two protocols, NFSv4 and CIFS, concurrently.

Now an admin wants to change an empty directory in this local file 
system to a junction.

The local FedFS admin server would then be required to recognize and 
translate either the CIFS form of the pathname of that directory, or the 
NFSv4 form of that pathname.

Does this then mean that the local FedFS admin service has to understand 
the CIFS share state and the local NFSv4 pseudofs well enough to 
translate either form of that pathname to get to the target directory?

What happens if that directory appears in multiple different places in 
the NFSv4 pseudofs, or is accessible via more than one CIFS share?  What 
happens if the local NFSv4 pseudofs is virtual, created by a user space 
process, and thus is not visible to any other process on the local system?

While the language of your suggestion is clear, I wonder how easy it is 
to implement it to the letter.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Mar 2010 12:06:23 -0500

On Mon, Mar 22, 2010 at 12:56:21PM -0400, Chuck Lever wrote:
> Suppose we have a file server that exports a given local file system
> via two protocols, NFSv4 and CIFS, concurrently.
> 
> translate either the CIFS form of the pathname of that directory, or
> the NFSv4 form of that pathname.

Note that what I'm proposing involves indicating which type of file
access protocol the given path uses.

> Does this then mean that the local FedFS admin service has to
> understand the CIFS share state and the local NFSv4 pseudofs well
> enough to translate either form of that pathname to get to the
> target directory?

Yes.  I don't think this is hard.

> What happens if that directory appears in multiple different places
> in the NFSv4 pseudofs, or is accessible via more than one CIFS
> share?  What happens if the local NFSv4 pseudofs is virtual, created
> by a user space process, and thus is not visible to any other
> process on the local system?

All of that is irrelevant because the path sent by the client will say
which kind of path it is, CIFS or NFSv4 (or whatever else).  Therefore
the ADMIN server will have the information it needs to canonicalize the
path.

> While the language of your suggestion is clear, I wonder how easy it
> is to implement it to the letter.

It'd be easier to put the ADMIN protocol in the file access protocol.
That would certainly avoid a variety of issues.  But it's also a
non-starter (we can't modify CIFS, and modifyin NFSv4 is a years-long
endeavour with the current train model).

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 22 Mar 2010 13:40:08 -0400

> the ADMIN server will have the information it needs to canonicalize the

Again, your suggestion is clear.  But I'm just not sure it will be easy 
in general (ie, on all O/Ses and for all interesting file protocols) to 
translate those pathnames in a user process separate from the mechanisms 
serving the file protocols, even if the file protocol type _is_ specified.

I'm interested in hearing from other implementers on this.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Mar 2010 12:49:44 -0500

On Mon, Mar 22, 2010 at 01:40:08PM -0400, Chuck Lever wrote:
> Again, your suggestion is clear.  But I'm just not sure it will be
> easy in general (ie, on all O/Ses and for all interesting file
> protocols) to translate those pathnames in a user process separate
> from the mechanisms serving the file protocols, even if the file
> protocol type _is_ specified.

Surely you agree that if the ADMIN protocol were an integral part of the
file access protocols then there should be no difficulty in
canonicalizing paths passed to it by the client.  Correct?

IMO the ADMIN protocol is properly viewed as an integral part of the
file access protocols, even if on the wire it is not.

In terms of Solaris, since that's probably where you're coming from, one
would have to consider a variety of approaches to implementing path
canonicalization in a user-land implementation of the ADMIN protocol.
These are plainly obvious, and almost certainly applicable to other
implementors:

 - Bake-in to the ADMIN server program the rules for converting from
   {file access protocol, path} to {local POSIX API path}.

   This cannot be hard, but would require that: a) such rules be static,
   b) changes to them be synchronized in a variety of places now
   including the ADMIN server program.

 - Provide a private system call by which the ADMIN server program can
   canonicalize such paths.

I can also think of other, less obvious approaches.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 22 Mar 2010 14:26:59 -0400

> canonicalization in a user-land implementation of the ADMIN protocol.

I'm providing a Linux implementation of the fedfs protocol.

>     including the ADMIN server program.

On Linux, as I understand it, the pseudofs is created out of whole cloth 
by mountd.  The Linux kernel calls up to mountd to discover the shape of 
the pseudofs as mount requests are processed.  Making the pseudofs 
visible to other user space processes and consistent with all views may 
be a challenge.

Other implementations provide file service entirely in user space. 
Isn't Samba a user space file service provider?  What about Ganesha?  In 
which case, how would a separate daemon perform pathname translation?

In some cases, NFSv4 service is not only a user process, but the user 
process also provides the local file system implementation.  ie, the 
local file system isn't even visible to other local processes or the 
kernel -- it must be accessed explicitly via NFSv4 loopback.

Certainly, translation is possible to do.  But why force it to be 
difficult?  Again, I'd like to hear from other implementers to see if 
this would be a problem.

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Mar 2010 13:55:29 -0500

> I'm providing a Linux implementation of the fedfs protocol.

Ok.

> On Linux, as I understand it, the pseudofs is created out of whole
> cloth by mountd.  The Linux kernel calls up to mountd to discover
> the shape of the pseudofs as mount requests are processed.  Making
> the pseudofs visible to other user space processes and consistent
> with all views may be a challenge.

You'd need an interface between the ADMIN server and mountd.

> Isn't Samba a user space file service provider?  What about Ganesha?
> In which case, how would a separate daemon perform pathname
> translation?

You'd also need an interface to Samba et. al.

> In some cases, NFSv4 service is not only a user process, but the
> user process also provides the local file system implementation.

In that case you lose.  But the problem isn't the sub-proposal we're
talking about.  The problem is the fact that the ADMIN protocol is
separate from the file access protocols.  That's a fundamental aspect of
the current ADMIN protocol proposal, and it makes it fundamentally
difficult _for some implementation designs_.

What we could do is have a different incarnation of the ADMIN protocol
(or different ports) for each file access protocol.  Then pure-user-land
servers can implement ADMIN internally.  I don't think I care to cater
to pure-user-land servers, but I don't object either.

> difficult?  Again, I'd like to hear from other implementers to see
> if this would be a problem.

I'm not doing that -- it's already done.

Nico
--

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Mar 2010 16:32:35 -0500

> to pure-user-land servers, but I don't object either.

There's other issues.  Suppose you have three NFS servers running on
different ports?  Then how can the ADMIN server know which one some path
is from?  You'd have to include not only the type of file access
protocol the path is for, but also the server instance (port number).

If those servers are all based on different pure-user-land
implementations then chances are that you will not be able to put
together a single ADMIN server implementation that can work for all of
them.  Instead you may need to run distinct ADMIN instances tightly
integrated with the corresponding file server processes, each instance
on a different port (or perhaps mediated by a switch front-end on one
standard port number).

I think that's a complication that we shouldn't care about because if we
did we'd never finish.  Why?  Because the only obvious solution is to
make ADMIN be a part of each file access protocol (or at least NFSv4).
That's clearly a huge political problem, even if we limit ourselves just
to NFSv4 -- even if we agree to dump the train model of NFSv4
extensibility and adopt an a la carte model instead, the work just to
create the a la carte scheme might take long enough to be equivalent to
"forever" from the perspective of an ADMIN would-be implementor.

So let's summarize:

 - a generic ADMIN server is problematic for not-very-tightly-integrated
   multi-protocol file server implementations;
 - a file access protocol-specific ADMIN server is also problematic;
 - an ADMIN server sub-protocol of file access protocols is also
   problematic.

What choice do we have but to pick our poison?

My preferences, in most-to-least preferred order:

#1: Dump the NFSv4 train model of extensibility for a la carte and focus
    on ADMIN as a sub-protocol of NFSv4.

    This one's rhetorical; I won't hold my breath.

#2: An HTTP-based RESTful ADMIN protocol.  This would allow us to put
    file server protocol and instance information in URLs, for example,
    and handle multiple ADMIN instances via redirects.  Among other
    things this would provide an off-the-shelf special character
    escaping scheme for pathnames.  HTTP/REST is all the rage, so one
    huge plus here is that we'd be fashionable :)

#3: An ADMIN protocol on one port with pathnames given as

    {file access protocol type ID,
     file access protocol instance (port number),
     path (opaque)}

    (that is, what we've been discussing).

#4: An ADMIN protocol just for NFSv4 and on port 2049 (i.e., different
    RPC program number, same TCPport number).  ADMIN for other file
    access protocols will have to be done separately for those other
    file access protocols.

#5: What reasonable options am I missing?

#6: Give up: no ADMIN protocol.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 22 Mar 2010 18:46:31 -0400

>      access protocols will have to be done separately for those other

If NFSv4.x is really as hard to change as you say, I like #4.

> #5: What reasonable options am I missing?

You might also consider an NFSv4 only admin service that takes a file 
handle instead of a pathname.  Such a service can be implemented on port 
2049 or some other port.

> #6: Give up: no ADMIN protocol.

For a "first pass" it would be simpler to include only NFSv4, but allow 
other file protocols to support junctions that are created by local 
tools on the server or via NFSv4.

---

From: James Lentini <[email-masked]>
Date: Mon, 22 Mar 2010 18:48:33 -0400 (EDT)

Replies below:

> returned in the fs_location* attributes.

Describing Alternative B is important for the reasons cited above. 
Backward compatibility for legacy clients is an important consideration 
and the option to support these configurations strengthens the 
solution.

> - SHOULD use alternative A

I don't know if the first MUST is necessary. I think the second and 
third statements would resolve any possible ambiguity (especially 
given the "strongly suggest" statement for alternative A that is 
already in the draft on page 8).

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Mar 2010 17:57:01 -0500

> If NFSv4.x is really as hard to change as you say, I like #4.

I wouldn't object.

> You might also consider an NFSv4 only admin service that takes a
> file handle instead of a pathname.  Such a service can be
> implemented on port 2049 or some other port.

I think that presents its own set of implementation problems.

> For a "first pass" it would be simpler to include only NFSv4, but
> allow other file protocols to support junctions that are created by
> local tools on the server or via NFSv4.

I agree.

Let's make ADMIN be NFSv4-specific and add CIFS and other support later.

Nico
--

---

From: James Lentini <[email-masked]>
Date: Mon, 22 Mar 2010 18:59:13 -0400 (EDT)

Replies below:

> >      (that is, what we've been discussing).

Over the long term, I think that this approach will prove the most 
flexible. Decoupling the administrative mechanisms from the file 
access protocol allows the two to evolve independently.
 
> If NFSv4.x is really as hard to change as you say, I like #4.

I don't see any significant difference between #3 and #4. The file 
access protocol type ID is always NFSv4 in #4.

> You might also consider an NFSv4 only admin service that takes a file handle
> instead of a pathname.  Such a service can be implemented on port 2049 or some
> other port.
>
> For a "first pass" it would be simpler to include only NFSv4, but allow other
> file protocols to support junctions that are created by local tools on the
> server or via NFSv4.

I'm not sure what you are suggesting as a "first pass". I'm 
interpreting "give up" to mean that no administrative operations 
would be defined. That would be an unfortunate outcome.

---

From: James Lentini <[email-masked]>
Date: Mon, 22 Mar 2010 19:01:34 -0400 (EDT)

> Let's make ADMIN be NFSv4-specific and add CIFS and other support later.

Take a look at the XDR structures I described yesterday. They define 
the path type field you've suggested and define a value for NFSv4.

-james

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 22 Mar 2010 18:06:17 -0500

> access protocol type ID is always NFSv4 in #4.

In #3 the ADMIN protocol runs on a port other than 2049.  Otherwise #3
and #4 and practically the same.

> would be defined. That would be an unfortunate outcome.

"Give up" means "we couldn't reach consensus so we dropped this and,
yes, we ended up with ad-hoc junction administration".  Indeed, that
would be most unfortunate.

However I think it very likely that we won't settle on #5.  I think I've
made a convincing argument for keeping ADMIN entirely NFSv4-specific for
now, and for either ignoring the possibility of multiple NFSv4 services
on different ports at the same addresses, or running the ADMIN protocol
on the same port as NFSv4.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Mon, 22 Mar 2010 20:08:04 -0400

> access protocol allows the two to evolve independently.

This approach (along with stipulating that the junction must be exposed 
via an export or share) does disambiguate the meaning of the path 
argument, but it still doesn't make it easy to convert the opaque path 
to a local file name.  Perhaps level of implementation difficulty is not 
a consideration at this point, but I tend to think it goes to how 
quickly such a protocol might be adopted.

So, #3 makes more sense to me than what is in the draft now.

> access protocol type ID is always NFSv4 in #4.

If the local file server provides both CIFS and NFSv4 services, for 
example, the problem, then, is how you might register the same admin RPC 
program and version on two different ports, one for CIFS and one for 
NFSv4.  So this solution almost requires that file protocol-specific 
admin services are not registered with the local portmapper, unless 
there is only one admin service.

And maybe it doesn't even make sense to run an RPC service on the CIFS 
file server port.

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 23 Mar 2010 01:49:42 -0500

On Mon, Mar 22, 2010 at 08:08:04PM -0400, Chuck Lever wrote:
> example, the problem, then, is how you might register the same admin
> RPC program and version on two different ports, one for CIFS and one
> for NFSv4.  So this solution almost requires that file
> protocol-specific admin services are not registered with the local
> portmapper, unless there is only one admin service.

You wouldn't.  Instead you'd define a brand new sub-protocol for CIFS to
do what ADMIN does.  The problem with this approach is that it's a
non-starter for any protocol other than NFSv4.

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 25 Mar 2010 12:08:30 -0400

Would everybody be happy with James's version, including the SHOULD and
MAY qualifiers to describe the location of SRV processing for an NFS
name space?

		Craig

-------------------------

> From: Lentini, James
> Sent: Monday, March 22, 2010 6:49 PM
> > > > > would be preferable to punt all resolution issues to the
> client.
> > I
> > > I'm not saying that SRV RRs are insufficient (they are
sufficient).
> > > Rob
> > > seems to be complaining that the server should not be the one
doing
> > the
> > I don't know what you're arguing.  I still believe that it's
possible
> to

---

From: Robert Thurlow <[email-masked]>
Date: Thu, 25 Mar 2010 17:05:02 -0600

>> already in the draft on page 8).

This works for me, Craig.

Thanks,
Rob T