Subject: [nfsv4] [FedFS] Meeting Minutes, 11/5/2009

From: James Lentini <[email-masked]>
Date: Thu, 5 Nov 2009 15:58:25 -0500 (EST)

FedFS Meeting Minutes, 11/5/2009
---------------------------------

Attendees
---------

Craig Everhart (NetApp)
Paul LeMahieu (EMC)
James Lentini (NetApp)
Renu Tewari (IBM)
Robert Thurlow (Sun)
Chris Stacey (EMC)
Nico Williams (Sun)

Minutes
-------

+ IETF Note Well Agreement

  This is a reminder that our discussions are governed by the 
  IETF Note Well Agreement. See:

    http://www.ietf.org/NOTEWELL.html

  We will start each week's meeting with this announcement.

+ Admin STAT command

  James re-capped last weeks discussion.

  We reviewed the concerns about how to report statistics 
  in an implementation independent function. 

  The results of this procedure would be error counters and 
  junction resolution counters. 

  As an aside, Paul envisioned a traceroute-style reporting tool
  being constructed. Such a tool would be useful when a user 
  told an administrator "I can't get to my data". The tools would 
  be used to determine where the error occurred. It might need to 
  traverse multiple junctions before locating the source of the 
  problem. There command would take the path as input, and trace 
  the path using the current set of FedFS operations. Paul did 
  not think additional operations (beyond the extensions to 
  FEDFS_LOOKUP_FSN described below) would be necessary.

  We then returned to discussing a STAT command.

  Craig raised the issue of information leakage and permissions
  for a  command since it would allow the administrator 
  to determine what the fileserver is doing on everyone else's 
  behalf.

  Rob suggested that if such an operation were added that it 
  be optional to implement.

  Chris speculated that such a facility might be useful for 
  diagnosing a Federation error. He envisioned a tool that would 
  examine all the fileservers in a namespace and give a yes/no 
  if anyone of them were seeing resolution errors. For each of 
  the fileservers experiencing errors, a second level of detail 
  would be provided on which junctions were failing.

  We need more information on the use cases for such a facility 
  before determining if it is necessary.

  [AI] Chris Stacey will write up example use cases for 
       a statistics command.

+ Admin FEDFS_LOOKUP_FSN resolve parameter

  We reviewed the updates posted here:

  http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admin-03bis.txt

  and diff here:

  http://jlentini.users.sourceforge.net/draft-ietf-nfsv4-federated-fs-admin-rfcdiff.html

  The first parts of the diff should be ignored. The xml2rfc tool has 
  been updated to place the abstract first in the document.

  The real changes start on page 6.

  Craig suggested adding an unstructured text field to the results to 
  indicate the source of a failure.

  We discussed the NSDB error. There is only one error being proposed without 
  distinction between NSDB node is down, the NSDB service is not listening, 
  the fileserver can't negotiate an LDAP connection, etc. There was consensus 
  that a coarse grained error for failure to establish and end-to-end NSDB 
  connection was all that should be provided.

  We also discussed reflecting LDAP error codes back through the procedure's 
  results. We agreed that the free form string field that will be added is 
  the appropriate place for this.

  Rob suggested that a malformed response be added as a fourth type of
  resolution error.

  As an aside, we discussed what the fileserver's behavior would be 
  when a junction resolution error occurred when a client was traversing
  the junction. What should the client see? Should the fileserver wait or 
  return an error? It seemed prudent for the fileserver to return a DELAY
  error if the problem was transient. For persistent errors the fileserver 
  should indicate a hard error.

  The admin protocol should return an error on junction resolution failure 
  immediately. This will allow the administrator to diagnose problems 
  immediately.

  Craig suggested that the resolve parameter be a ternary instead of binary
  value: 0 = don't resolve, 1 = resolve using cache value, and 2 = resolve 
  using nsdb.

  We also agreed that a generic catch all for resolution errors would be 
  useful to distinguish a resolution error from SVRFAULT.

  [AI] James Lentini add malformed response error code, string field to 
       response, generic resolve error code, and make resolve a 
       ternary value.

+ Admin NSDB Trust Anchor Management  

  Nico proposed TA format and TAMP, pick one as required to implement. 
  We need to discuss this more.

  [AI] James Lentini to discuss TA management with Nico.

+ No Meeting Next Week

  Next week is IETF'76 in Hiroshima. We will not meet next week.

---

From: Nicolas Williams <[email-masked]>
Date: Thu, 5 Nov 2009 15:25:50 -0600

>   indicate the source of a failure.

Beware of unstructured text fields!  You can't localize them _unless_
you have the client send language tags so the client and server can
negotiate a language to localize to.  See RFC5646, a.k.a. BCP0047.

For G11N (globalization) reasons you must support localization...

If you can come up with a complete list of errors, then it's better to
have error codes for all of them.  You can always punt and say "go look
at logs" (logs are never localized).

>   connection was all that should be provided.

A coarse error is a lot better than nothing.  But there's no reason that
we couldn't specify a finer-grained set of errors and let
implementations use them that can.  The errors I can think of:

 - NSDB name NXDOMAIN (not resolvable)

 - NSDB name resolution timeout (e.g., DNS servers not reachable)

 - NSDB name partially resolvable (e.g., can find SRV RRs but not A
   RRs; this would be misconfiguration)

 - NSDB hosts all not reachable / connect timeout

 - NSDB hosts' server certificates cannot be validated

 - NSDB LDAP SASL/GSSAPI bind failed (should probably have some
   sub-errors for this)

    - Unfortunately the set errors for this could be enormous, and
      because SASL and the GSS-API are open plug-in systems,
      indeterminate.

      Here an unstructured text field would be nice.  (Unfortunately at
      least one very common LDAP library does a very poor job of
      reporting SASL/GSSAPI bind errors, and this is probably true of
      other LDAP client implementations.)

 - NSDB LDAP protocol error N

 - NSDB object not found

 - NSDB object missing required attributes

>   the appropriate place for this.

See above.

>   should indicate a hard error.

How long to delay?  Users will want to know.  Shouldn't NFSv4.1 have a
way to say "this is a junction, but I can't figure out where to refer
you to"??

>   immediately. This will allow the administrator to diagnose problems 

Sure.

>   value: 0 = don't resolve, 1 = resolve using cache value, and 2 = resolve 

What cached value?

>        response, generic resolve error code, and make resolve a 

Nico
--

---

From: James Lentini <[email-masked]>
Date: Fri, 6 Nov 2009 09:08:15 -0500 (EST)

> at logs" (logs are never localized).

Thank you for making us aware of that. Including a text field would be 
in the "bells and whistles" category. If it would require a 
significant increase in complexity, it probably isn't worth it.
 
>  - NSDB LDAP protocol error N

We could add more error codes. They won't eliminate the need for an 
administrator to diagnose the problem. 
 
>  - NSDB object not found

The FEDFS_ERR_NOFSN and FEDFS_ERR_NOFSL error codes in the update are 
for this case.

>  - NSDB object missing required attributes

Rob suggested the same thing below (see note about malformed response 
error). That is on the list of error codes to add.

> way to say "this is a junction, but I can't figure out where to refer

This was a side discussion. The NFS GETATTR operation, which is used 
to fetch the fs_locations or fs_locations_info data, may return the 
error NFS4ERR_DELAY. We were discussing times when it would be 
appropriate for the fileserver NFS4ERR_DELAY. Obviously, this isn't 
part of FedFS. It has already been described in NFSv4.

> >   value: 0 = don't resolve, 1 = resolve using cache value, and 2 = resolve 

The fileserver may have cached FSL values. See Section 2.4.2 "Caching 
of Fileset Locations" in 

http://www.ietf.org/id/draft-ietf-nfsv4-federated-fs-protocol-04.txt

---

From: Nicolas Williams <[email-masked]>
Date: Fri, 6 Nov 2009 10:49:33 -0600

> significant increase in complexity, it probably isn't worth it.

I agree, it's bells and whistles.

> administrator to diagnose the problem. 

Indeed, but it's very nice to go in to battle with a clue.

Looking at this from a user-friendliness p.o.v., where the user is a
sysadmin, but still a user, a human being, one who will curse our names
if we make life too hard for them...

...well, the implementation will have known what went wrong, so why not
tell the sysadmin?  Now, I grant you that when you have many layers it's
easy to lose track of specific errors and surrounding context, but just
because that's a common result doesn't mean this protocol should carve
that result into stone.

> The FEDFS_ERR_NOFSN and FEDFS_ERR_NOFSL error codes in the update are 

OK.

> error). That is on the list of error codes to add.

OK.

> part of FedFS. It has already been described in NFSv4.

I understand, and I think that's unfortunate.  The NFSv4 server could
distinguish a bit more and say "this is a junction, but I can't resolve
the referral for it".  Compare to "I may need to pull this from a
tape library, hold on a minute" -- clients might react differently.  Oh
well.

> The fileserver may have cached FSL values. See Section 2.4.2 "Caching 

Should we have a notion of FSN/FSL soft and _hard_ TTLs?  Soft TTL
expiration -> attempt to re-resolve.  Hard TTL expiration -> if can't
resolve, then the junction returns an error.

Nico
--

---

From: James Lentini <[email-masked]>
Date: Sat, 7 Nov 2009 09:01:10 -0500 (EST)

> resolve, then the junction returns an error.

The current TTL field is what you would describe as a "hard TTL". The 
question therefore is if a soft TTL field should be added. At a 
glance, it doesn't seem necessary since the fileservers are free to 
implement such a policy on their own. I'm concerned that adding an 
additional TTL field increases the complexity of the administration 
model. More knobs and dials increase the chance for a configuration 
error. I'd argue that the fileserver software is in the best position 
to make intelligent decisions about refreshing its FSN-to-FSL cache.