Subject: RE: Issues list for RFC3010

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 21 Sep 2001 09:24:14 -0700

Spencer Shepler wrote: 
> On Thu, Jeff A. Smith wrote:
> > "Khan, Saadia" wrote:
> > > 
> > > -----Original Message-----
> > > I would like to add some more things to the list which are not very clear as far 
> > > as the spec goes:
> > > 
> > > Named Attributes:
> > > 
> > > 1. Does it make sense to allow setattr on a named attribute dir? Is that
> > > allowed in the spec? And if so what does it mean if the named attr dir has 
> > > different perms, uid, gid from the base file and in that case which one is 
> > > used for permission checking?
> > > 
> > > In similar context, does it make sense to do the same for named attributes? 
> > > Or does changing perms, uid, gid for a named attribute change the perms, uid, 
> > > gid for the base file.
> > > 
> > 
> >         As I understand it, the spec provides a way to get a filehandle for the 
> >         attribute directory with OPENATTR, but it does not restrict the how that 
> >         filehandle can be used.  The underlying server fs may restrict the set of 
> >         operations allowed on attrdirs (for reasons you mentioned), but the spec 
> >         does not.  I think this general answer addresses most of your questions.
>
> As Jeff mentions, the named attribute support was intentially under
> specified.  I believe the things you ask about could be allowed for
> certain implementations but I doubt they would not be very useful.  My
> preference is to leave this type of functionality under specified and
> allow client named attribute APIs and server implementations drive
> further definition while not restriction the protocol to a particular
> implementation.

I know this was intentional, but is it the right thing to do?

Leaving this under-specified opens the way for interoperability
problems which are going to seriously compromise the usefulness
of named attributes, unless somehow there is arrived at a minimal set 
of behaviors that clients expect and that all servers that support
named attributes implement.  If that's defined, then we can let
the lunatic fringe be the lunatic fringe and go on.  If we don't
do this in the spec, I suppose we can do it over a beer at the next
bakeoff.  But if we do that, we should let people who aren't there 
know what we're intending to do.

Some of Saadia's questions deal not with the semantics of the named
attributes themselves but of the named attribute directory itself.
The named attribute directory may not really exist.  It is an 
artifact to allow named attributes to be (relatively) easily fit 
into NFS's file operation paradigm.  I'd argue that there is 
a difference between leaving something under-specified and leaving
it way under-specified.  I can see that there might be interest
in a different and richer named attribute models (I'll admit it is 
possible that the lunatic fringe is really just ahead of it's
time).  I can't see the same possibility for operations on the named
attribute directory itself.  I would argue that SETATTR on the
named attribute directory is an operation that clients should not
use and servers should not implement even if their implementation
(such as making the named attribute directory a more-or-less
normal directory would allow it).  I think we'd all be better off 
if the spec said so.

With regard to Saadia's other question, about setting attributes
on named attributes themselves, I would argue that it's best for
the server not to deceive.  It may (and probably will) restrict
the set of attibutes that it keeps for named attributes but if 
the client asks it to set one that it doesn't keep separately, it
should reject the request rather than set the corresponding 
attibute on the base file.

> I will add an item to the issues list to discuss the possible
> implementations and pitfalls of such.

> > > 2. Currently a server is required to first check if a named attribute dir
> > > associated with a file/dir actually contains any named attributes before
> > > returning TRUE in the named_attr field for getattr.
> > > 
> > > There can be a couple of issues with that. First its very improbable to have 
> > > a named attr dir without any named attrs, and second it is possible that a 
> > > client is requesting the server for all supported attributes every time it 
> > > does a stat call. It can be very expensive for the server to check if the 
> > > directory is empty or not everytime it gets one of these calls.
> > > 
> > > It should be sufficient for the server to check if the dir exists and return 
> > > TRUE  for named_attr.
> > > 
> > 	In this case, the spec is trying to be general enough to provide useful
> > 	information to applications without assuming a particular server fs 
> > 	implementation.  I know of at least one server fs where empty attr dirs 
> > 	are not rare.  The attrdirs in this file system are like regular dirs 
> > 	in that they are not removed when the last named attribute is removed.  
> > 	I'm not saying I think that design is correct or that I like it -- I'm 
> > 	just saying it exists.  
> > 
> > 	If you believe that it is reasonable for an app to set several 
> > 	attributes and clear them later, then empty attr dirs may not be such
> > 	a rare occurrance in the server file system.  For such a file system, 
> > 	checking to see that the attrdir exists would not provide very useful 
> > 	info to the app -- especially if all file system objects had a named 
> > 	attribute at some point in their lifetime.

I've got to believe that an environment in which all file system objects
have a named attribute during their lifetime but which a large number 
don't have one at any given time, is likely to be one in which the file
system would get rid of empty attribute directories automatically.

> > 
> > 	I'm not sure I agree with your first statement that servers
> > 	must check to see that the attrdir actually contains attributes
> > 	before returning TRUE for named_attr.  The server must answer
> > 	the question correctly, but it might not have to do readdir
> > 	to do so.  If you're writing a nfsv4 server, and all server filesystems 
> > 	remove attrdirs after the last attribute is deleted, (and 3rd party
> > 	filesystems are not possible), then you can certainly code your 
> > 	NFSv4 server to take advantage of that and simply check for the 
> > 	existance of the attr dir instead of doing readdir.

I think Saadia was thinking of cases in which the server does not 
automatically remove empty named attribute directories.

In such an environment, it is most probable that if a given object has
a named attribute directory it has some attribute within it but that
is not assured.  So in that case, the server, to satisfy a GETATTR 
would have to look in the directory, to determine whether it is 
empty and then when it turned out it wasn't empty, the client would
typically issue an OPENATTR/READDIR to actually find the attributes
so that the server would wind up looking in the directory twice.
 
> 
> I don't expect that clients will ask about named attributes that much
> in the general case.  We could add language that discourages the
> overuse of the named_attr attribute if that would help.

There may be a large number of clients, at least initially, that don't
care about named attributes at all, but I'm concerned only about the
ones that do.

How would you define overuse?  I think the purpose of the named_attibute
attribute is so that the client can know whether he should bother
doing the OPENATTR/READDIR when it is of no use.  In an environment in
which there are no empty named attribute directories possible, the server
can trivially detremine this.  In an environment where empty ones can exist,
the server currently must check even when this is very unlikely to save the
client work.

So I would argue that the purpose of named_attr is as an optimization 
and that it should be defined so that,

      If there are named attributes the server must return TRUE.

      If the server can easily determine that no named attributes
      exist, the server should return FALSE.

      If the server determines that it is likely that named attributes
      exist but this is not guaranteed the server may return TRUE 
      and the client will find out that the set of named attributes
      is NULL when it does OPENATTR/READDIR.

> > > 3.Should exclusive create be supported for named attributes?
> > >
> > 
> > 	I don't see any reason why it shouldn't, and the spec doesn't specifically
> > 	disallow it, so I would say yes.  I don't have a strong argument for either
> > 	position, and I generally dislike arbitrary restrictions...  This sounds like 
> > 	a good thing to be explicitly stated in the spec.
> 
> Since we have taken the model that general operations apply named
> attributes, the answer would be yes.  However, with the case of
> exclusive create, we need to be sure that this is something that is
> applicable in the implementations that are planned.  Would like to
> avoid specifying something that will not be implemented and removed
> from the specification later on. :-)

In the particular case of exclusive create, the model is that the
create verifier is stored where the attributes normally are and that
a SETATTR done after.  Both of these may present difficulties among some
of the myriad possible implementations.  So whatever the spec says, the
real question is whether cleints are going to do this.  If they do,
server implementations are going to be constrained.  On the other hand,
I can see that this might be a useful facility for the client.

I'm intrigued by your remark about implementation applicability and
the process of going to Draft Standard (I'm not sure how far back your
smiley was intended to cover).  If this rule would apply this way
to exclusive create of named attributes, why wouldn't it also apply
to create of a directory, symlink, or device in a named attribute
in a named attribute directory, openattr-openattr-openattr ad
infinitum, and all the other stuff that I'm currently recoiling from
in horror?  It's unlikely that there will two inter-operating
implementations of a lot of those things.

> > > 4. Does it make sense to rename a named attribute? I dont think it makes
> > >    sense to rename a named attribute between different base files. Within 
> > >    the same file, maybe but even that doesn't make much sense.
> > > 
> > 	Again I don't really have a position, and the spec is general enough
> > 	to support implementations that support renaming attributes as well
> > 	as those that don't.  Perhaps named attrs should be mentioned in the
> > 	"rename" implementation section of the spec.
> > 
> > 	I don't have any experience coding apps that use named attrs, so my
> > 	$0.02 is more like $0.005, but one use of renaming attributes would be
> > 	to enable atomic set (create named attr with a temp name, write all
> > 	attr data, then rename it to whatever the "real/proper" name).
> 
> Right. This is the example that I have always thought of.  Since there
> are named attribute APIs that require atomicity with respect to
> setting a value of the named attribute and NFSv4 does not provide that
> type of atomicity, a WRITE/RENAME sequence could be used in its place.

> For the case of rename of a named attribute between filesystem
> objects, I believe that is unlikely and the server can return
> NFS4ERR_XDEV in this case (and probably should).  I will add this
> clarification to the RENAME operation IMPLEMENTATION section.

RENAME has lots of interesting cases like rename of a named attribute
to be a regular file and vice versa.  LINK also has interesting
possibilities.  How about NFS4ERR_WAY_TOO_WEIRD? :-)

> > > 5. Does the spec support deleting the named attribute dir? I think that 
> > > deleting a file with named attributes should delete everything with it, 
> > > including the dir and all named attributes or the client should be allowed 
> > > to delete a single attr.
> > > But allowing a named attr dir to be deleted without deleting the base file 
> > > doesn't really make much sense.
> > 
> > 	Because the attrdir has no name and the spec doesn't define a
> > 	RM_ATTR_DIR op, there's no way to explicitly delete the attrdir.
> > 	The only way I know of to delete the attrdir is to implicitly 
> > 	delete it by the deleting the base file.  This is also a good 
> > 	candidate for clarification in the spec.  
> > 
> > 	Of course, a server fs implementation would be free to delete the 
> > 	attrdir after the last attribute was removed.  So, it could be 
> > 	possible for the attrdir to be removed without removing the 
> > 	"base file" -- it depends on the server fs implementation.  I
> > 	think it would be totally broken to be able to remove a file 
> > 	without	also removing its attributes.

> Jeff, did you mean to say something like, when a file is REMOVEd the
> named attributes associated with that file should automatically be
> removed (implicitly).  It isn't the responsibility of the client to
> first remove all of the named attributes before removing the
> filesystem object.  This is different from removal of a regular
> directory that is non-empty. 

This is one problem with the "Named attribute directories are
exactly the same as any other directories except where required
by the nature of named attributes"-model.  It is hard to 
determine when that is.

I have another question about what Jeff was saying.  I had been 
assuming that a "server fs implementation is free to delete the
attrdir after the last attribute was removed" only if the handles
that it created for named attribute directories were such that
they would not go stale simply because a named attribute was deleted.

To use implementation terms, 

If you have handles for named attribiutes directories that have the 
inode number of the base file plus a flag then you can automatically
create the directory when the named attribute is created and delete
it when it is empty.

However, if you have handles for named attributes that have the inode
number of the named attribute directory, then you can't automatically
delete when the last named attribute goes away sicne this would make
the handle returned by an earlier OPENATTR stale.  Are you saying that
you can do the autommatic deletion in this case as well, Jeff?

> The key in the NFSv4 named attribute support is that it allows for
> various implmentations.  It is possible that the attribute directory
> returned from OPENATTR doesn't have a corresponding directory at the
> server.  The server may choose to implement named attributes in a way
> that really doesn't have a physical directory in the server
> filesystem.

Multiple implementations are good.

Multiple interfaces are not so good.  

I'm worried that the current approach allows so much variation
in the operations supported, that it may really wind up turning
into multiple interfaces expressed in a common language rather than
a single interface.

> The specification also allows for named attributes of named
> attributes.

"But do they go away at Draft Standard if nobody implements them?",
he asked with baited breath.

> I realized via this discussion that I had missed an issue related to
> named attributes.  We had discussed on the WG alias the addition of a
> CREATE flag to OPENATTR to provide for server implementations that do
> not create a named attribute directory until the first named attribute
> is actually created.  Therefore, if the client is just reading the
> attribute directory without the intent of creating a named attribute
> it can used OPENATTR() and the server can return NFS4ERR_NOENT.  If
> the client is going to create a named attribute, then it can use
> OPENATTR(CREATE) to have the named directory activated.
> 
> I will add this to the issues list.

I'm pretty sure I saw that in the list already but I don't remember the
number.

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 17:38:00 -0500

> know what we're intending to do.

Point taken.  So maybe what we should strive for in the case of the
named attribute support is a minimum of specification to allow for
interoperability with the maximum amount of flexibility for future
implementations.

> normal directory would allow it).  I think we'd all be better off 

Okay. So the proposal is to state that for the attribute directory (as
returned by OPENATTR) is not an appropriate target for SETATTR.  I
would propose that GETATTR has limited meaning as well for this
directory.

> attibute on the base file.

I agree and that has been my interpretation.  I will add an item to
provide this clarification as well.

> system would get rid of empty attribute directories automatically.

Seems reasonable.  No telling what will actually be implemented.

> so that the server would wind up looking in the directory twice.

We have a pathconf() api that allows the application to ask the
question of whether named attributes exist for a particular filesystem
object.  If there is no protocol support for this question, then the
client will have to do the OPENATTR, READDIR sequence itself.  Is that
a reasonable overhead for something like pathconf()?  Obviously, an
application that asks that question will more than likely want to
enumerate the named attributes so the READDIR would fill a local cache
potentially but it seems high overhead for a simple question.

> care about named attributes at all, but I'm concerned only about the

Well, Solaris, Linux, and the Windows clients will have support for
named attributes.  Maybe what you meant was that there won't initially
be that many applications that use named attributes. :-)

>       is NULL when it does OPENATTR/READDIR.

Well, in the case of the pathconf() example I provided, the client
will end up doing the OPENATTR, READDIR sequence to answer the
pathconf() question correctly (since its definition is not as lenient).

What you have proposed can be accomplished by just having the client
do an OPENATTR without the proposed CREATE flag.  If no filehandle
comes back, no attributes exist.  If a filehandle comes back, the
client does a READDIR to see if any attributes exist in reality.
Again, this seems like high overhead for the simple question at the
client of "named attributes?".

> implementations of a lot of those things.

Well, we could take it to that extreme for a move to Draft but I would
think that exclusive create is so fundamental in this instance that we
would want implementation experience.  

> possibilities.  How about NFS4ERR_WAY_TOO_WEIRD? :-)

or NFS4ERR_DEV_your_request_to_black_hole...

> you can do the autommatic deletion in this case as well, Jeff?

I think we can make the client recover but is that necessary.  I don't
know.  Maybe we should restrict the server's activities with respect
to this in that once a filehandle is returned as a result of OPENATTR
it is not supposed to go STALE until the associated object is removed.

> into multiple interfaces expressed in a common language rather than

So through the implementation experience we are gaining we can work to
place the appropriate framing around the current definition to make it
useful.

> he asked with baited breath.

My opinion is, yes, but maybe we should wait until we actually get
there before worrying about it too much.

> I'm pretty sure I saw that in the list already but I don't remember the

Well, I added it again if that was the case. :-)

-- 

- Spencer -