Subject: [nfsv4] [FedFS] meeting agenda (11/19)

From: James Lentini <[email-masked]>
Date: Wed, 18 Nov 2009 11:19:06 -0500 (EST)

Tomorrow's proposed agenda is below. Please let me know if you would
like to add additional topics.

    Time: Thursdays 1:30-2:30 PM EST/10:30-11:30 AM PST
 Dial-in: 1-888-765-3653 (us)
          303-218-2659 (international)
      Id: 2354843

Proposed Agenda
---------------

+ IETF Note Well Agreement

  This is a reminder that our discussions are governed by the 
  IETF Note Well Agreement. See:

    http://www.ietf.org/NOTEWELL.html

  We will start each week's meeting with this announcement.

+ Review Action Items

  - Chris Stacey will write up example use cases for 
    a statistics command.

  - James Lentini add malformed response error code, string field to 
    response, generic resolve error code, and make resolve a 
    ternary value.

  - James Lentini to discuss TA management with Nico

+ Discuss Update to FEDFS_LOOKUP_FSN procedure

  The FEDFS_LOOKUP_FSN definition was updated with 
  additional error values, including an option for 
  communicating LDAP error values back to the administrator, 
  a malformed response error value (FEDFS_ERR_NSDB_RESPONSE) 
  and a generic resolution error (FEDFS_ERR_NSDB_FAULT). 
  The resolution options include no resolution, cached, 
  and NSDB.

  The updated draft is here:

  http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admin-03bis.txt

  and a diff against the previous draft is here:

  http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admin-rfcdiff.html

+ Admin NSDB Trust Anchor Management

+ No Meeting Next Week

  11/26 is Thanksgiving.

---

From: "LeMahieu, Paul" <[email-masked]>
Date: Thu, 19 Nov 2009 10:07:53 -0800

James,

I have a few questions on the SRV record mechanism we talked about long ago, clarifying client behavior and also looking at what would happen with an NFS URL.

This is very much a non-critical topic, so only if we have the time.

--Paul

On 2009-Nov-18, at 8:19, James Lentini wrote:

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: <[email-masked]>
Date: Thu, 19 Nov 2009 13:30:36 -0500

I thought I'd hit send on this a while back.  Sorry for the delay.

We were talking about maintaining statistics about errors associated
with junction resolution and a standard way to extract those statistics
from each file server in a federation...

What prompted me was thinking about how to provide a namespace health
check tool that would give an administrator a traffic light view of the
health of a namespace and help them troubleshoot any issues.   Such a
tool would get a list of the file servers in a federation and query each
one to see if it has had any issues resolving junctions recently.   This
would allow the health check tool to provide a traffic light style
status of the namespace and federation members something like the
following: 

Status for namespace "ABC": 
- Green (all file server members up and no junction resolution failures
reported since last check)
- Yellow (all file server members up but at least one member reporting
junction resolution failures)
- Red (at least one file server member un-contactable, contactable
members may or may not be reporting junction resolution failures)

Status for each file server in federation:
- Green (no junction resolution failures reported since last check)
- Yellow (some resolution failures reported since last check)
- Red (un-contactable)

The natural next step in that the administrator will want to be able to
do is find out from any file servers reporting junction resolution
failures how many failures are occurring, which junctions are failing to
be resolved and why.   

I think figuring out why a given junction is having problems is
addressed by the procedures that James has added to the admin draft last
time we met.

I expect every file server implementation will maintain statistics about
how many junction resolutions it performs, including how many fail.
Just like we track how many reads, writes, get attrs, etc. we process
today.   So, this information will most likely be available.

The part that is missing is a standardized way to query each file server
for a junction resolution error count and list of junctions that it is
having trouble resolving.  Having a standard way to get this information
from each file server  would be very helpful.   It will be much harder
to implement such a health checking tool that can cope with a namespace
made up of file servers from multiple vendors without this because the
tool would have to detect the type of each file server and know how to
extract the required information from it.  

There are data leakage issues here.  If a file server is hosting file
systems that contain junctions from multiple namespaces the junction
resolution error count should be maintained per namespace otherwise an
error in a single junction will turn the status of all namespaces
yellow.   Similarly when querying for a list of junctions that a file
server is having trouble resolving, we want a list per namespace.  We do
not want to show the administrator for one namespace a list of junctions
from another namespace which they may not have any rights to or even
knowledge of.

Thoughts?

Thanks,
Chris

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of James Lentini
Sent: Thursday, 19 November 2009 5:19 a.m.
To: [email-masked]
Subject: [nfsv4] [FedFS] meeting agenda (11/19)

Proposed Agenda
---------------

+ Review Action Items

http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admi
n-03bis.txt

http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admi
n-rfcdiff.html

+ No Meeting Next Week

  11/26 is Thanksgiving.

---

From: James Lentini <[email-masked]>
Date: Fri, 20 Nov 2009 14:37:31 -0500 (EST)

On Thu, 19 Nov 2009, LeMahieu, Paul wrote:

> I have a few questions on the SRV record mechanism we talked about 
> long ago, clarifying client behavior and also looking at what would 
> happen with an NFS URL.

Paul,

Could you elaborate a bit on the two questions above?

In draft-ietf-nfsv4-federated-fs-dns-srv-namespace, the client is the 
DNS client, which is either an NFS client or an NFS server. Are you 
are asking about one or both of those?

Are you referring to NFS URLs as specified in RFC 2224? What specific 
interactions are you interested in?

-james

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 24 Nov 2009 10:40:16 -0600

> status of the namespace and federation members something like the

The main problem lies in the "get a list of" part.  Whether the things
to be listed are namespaces, junctions or fileservers in a domain, it
isn't easy, and in some cases it isn't at all feasible.

To list fileservers in a domain... you can't.  The best you can do is
list servers named in a given NSDB, and though servers should only be
mentioned in one NSDB, there's no guarantee of that.

To list junctions in a fileset... may require doing a recursive find
operation if the fileserver doesn't track junctions separately from the
filesystem structure.

To list namespaces... you can't.  The best you can do is put together a
list of namespaces referenced from junctions in filesets that you know
about.

If you have a separatly maintained list of things to monitor, then you
are in reasonably good shape.

> Status for namespace "ABC": 

Namespace or NSDB?

> - Yellow (some resolution failures reported since last check)

Remember too that creation of FSLs and FSNs in NSDBs is shoot-and-
forget.  You create the things, and you might never bother deleting
them, even if the actual filesets and the junstions referring to them go
away.  There's going to be a lot of garbage in NSDBs, and often only
heuristic ways to differentiate garbage from live entries.

What we can do is have servers report NSDB lookup error information in
useful ways:

 - NSDBs not resolvable
 - NSDBs not reachable
 - NSDBs not responding
 - NSDBs that can't be authenticated
 - NSDBs with other errors (e.g., can connect, authenticate, but then
   times out)
 - FSN objects not found
 - FSN objects with no FSL objects found
 - FSN objects with no resolvable FSL objects found
 - FSL objects not found

You might also get a list of FSLs resolved from junctions, so that you
could then go check if the filesets are all still there.  But that's
about it, I think.

> failures how many failures are occurring, which junctions are failing to

Yes.

> today.   So, this information will most likely be available.

But we have to say it must/should be available, and we should provide a
standard protocol for getting at it.  The ADMIN protocol seems like the
perfect place for it (is anyone interested in using SNMP for this?).

> extract the required information from it.  

Yes.

> from another namespace which they may not have any rights to or even

I don't think we should make it a requirement that junctions be
listable.  It'd be nice, sure, but it shouldn't be a requirement.  I
don't think servers should track failed junctions indefinitely either.
A fixed-sized circular log will do.

Given a log that includes enough information to find the absolute path
(from the server's pseudo-root) to each logged junction, the server can
check whether the ADMIN user has authorization to resolve each given
junction, and thus filter the logs accordingly.  That would be a slow
operation though!  I think servers should be allowed to implement
alternative authorization schemes such as: "ADMINs get to see all
junctions" and "ADMINs get to see all junctions subject to ACLs/MAC but
excluding path to junction lookup authorization checks".

Nico
--

---

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 24 Nov 2009 12:21:33 -0500

Nico as usual makes me think. 

> From: Nicolas Williams [mailto:[email-masked]] 
> The main problem lies in the "get a list of" part.  Whether 
> the things to be listed are namespaces, junctions or 
> fileservers in a domain, it isn't easy, and in some cases it 
> isn't at all feasible.
> 
> To list fileservers in a domain... you can't.  The best you 
> can do is list servers named in a given NSDB, and though 
> servers should only be mentioned in one NSDB, there's no 
> guarantee of that.
> 
> To list junctions in a fileset... may require doing a 
> recursive find operation if the fileserver doesn't track 
> junctions separately from the filesystem structure.
> 
> To list namespaces... you can't.  The best you can do is put 
> together a list of namespaces referenced from junctions in 
> filesets that you know about.
> 
> If you have a separatly maintained list of things to monitor, 
> then you are in reasonably good shape.

Ditto to all of these.  We don't have a requirement of any form for a
fileserver to have any efficient means of enumerating junctions from a
given fileset, except by enumeration.  And it's perfectly legal, if
maybe cumbersome in practice, for a domain/namespace/whatever, even a
single vendor's slice of it, to use multiple NSDBs for distinct
junctions.
 
> > - Green (all file server members up and no junction resolution 
> > failures reported since last check)
> Remember too that creation of FSLs and FSNs in NSDBs is 
> shoot-and- forget.  You create the things, and you might 
> never bother deleting them, even if the actual filesets and 
> the junstions referring to them go away.  There's going to be 
> a lot of garbage in NSDBs, and often only heuristic ways to 
> differentiate garbage from live entries.

I'm not sure this is germane to the idea of how file servers are able to
use their NSDBs.  But I don't see the NSDB as being such a magnet for
garbage.  Clearly it *can* be used that way, but it can also be used by
the admin as a reasonably authoritative repository of the fileset
instances in existence and under some management.

> What we can do is have servers report NSDB lookup error 
> information in useful ways:
> 
>  - NSDBs not resolvable
>  - NSDBs not reachable
>  - NSDBs not responding
> You might also get a list of FSLs resolved from junctions, so 
> that you could then go check if the filesets are all still 
> there.  But that's about it, I think.
> 
> > The natural next step in that the administrator will want 
> to be able 
> > to do is find out from any file servers reporting junction 
> resolution 
> > I expect every file server implementation will maintain statistics 
> > about how many junction resolutions it performs, including 
> how many fail.
> But we have to say it must/should be available, and we should 
> provide a standard protocol for getting at it.  The ADMIN 
> protocol seems like the perfect place for it (is anyone 
> interested in using SNMP for this?).
> 
> > The part that is missing is a standardized way to query each file 
> > server for a junction resolution error count and list of junctions 
> > that it is having trouble resolving.  Having a standard way 
> to get this information
> > to implement such a health checking tool that can cope with a 
> > namespace made up of file servers from multiple vendors 
> without this 
> > because the tool would have to detect the type of each file 
> server and 
> > know how to extract the required information from it.
> 
> Yes.
> 
> > server is having trouble resolving, we want a list per 
> namespace.  We 
> > do not want to show the administrator for one namespace a list of 
> > junctions from another namespace which they may not have 
> any rights to 
> > or even knowledge of.
> 
> I don't think we should make it a requirement that junctions 
> be listable.  It'd be nice, sure, but it shouldn't be a 
> requirement.  I don't think servers should track failed 
> junctions indefinitely either.
> Given a log that includes enough information to find the 
> absolute path (from the server's pseudo-root) to each logged 
> junction, the server can check whether the ADMIN user has 
> authorization to resolve each given junction, and thus filter 
> the logs accordingly.  That would be a slow operation though! 
>  I think servers should be allowed to implement alternative 
> authorization schemes such as: "ADMINs get to see all 
> junctions" and "ADMINs get to see all junctions subject to 
> ACLs/MAC but excluding path to junction lookup authorization checks".
> 
> Nico
> --

---

From: Nicolas Williams <[email-masked]>
Date: Tue, 24 Nov 2009 11:51:58 -0600

> instances in existence and under some management.

Garbage will happen.  A garbage collection heuristic would be useful;
something like:

    for each FSL in $nsdb
	if dns_lookup(FSL->server) == NXDOMAIN
	    insert(garbage_fsl_list, FSL)

    for each FSN in $nsdb
	is_garbage=true
	for each FSL in FSN
	    if dns_lookup(FSL->server) != NXDOMAIN
		is_garbage=false
		break
	    else
		is_garbage=true
	if is_garbage
	    insert(garbage_fsn_list, FSN)

But this can be implemented given nothing more than an LDAP client
(well, depending on how we resolve path encoding).

What do you think re: Stacey's comment about bad junction log access
controls?

Nico
--

---

From: James Lentini <[email-masked]>
Date: Tue, 24 Nov 2009 13:58:18 -0500 (EST)

> On Thu, Nov 19, 2009 at 01:30:36PM -0500, [email-masked] wrote:
<snip>
> perfect place for it (is anyone interested in using SNMP for this?).

There is interest in using SNMP for this. Chris Stacey is going to 
write up some thoughts on this topic. See the minutes from last week's 
call:

http://www.ietf.org/mail-archive/web/nfsv4/current/msg07564.html

---

From: "Everhart, Craig" <[email-masked]>
Date: Tue, 24 Nov 2009 14:03:05 -0500

> ACLs/MAC but excluding path to junction lookup authorization checks".

If it's part of the admin protocol, then perhaps the admin for a given
namespace could be able to view a log of recent problem junctions and
the coded errors encountered in resolving them, for the namespace being
administered.  An admin gets to administer.

But what can go into a junction record?  The file server won't,
generally, have a pathname ready to be logged.  The server will have an
FSN that it tried to evaluate and the errors accreted in evaluating it,
which is fine.  And it will have a filehandle for the junction object or
maybe its parent directory, but this won't be particularly useful (or
interpretable) in a generic tool.

		Craig