Subject: Re: [nfsv4] XATTRs in NFS?

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Oct 2013 13:44:25 -0700

Hi Spencer, 
Is it still possible to get on the agenda fir IETF 88, I just got approval 
to travel. We can use about 15 minutes, or what ever is available to 
discussion the future of named attributes in NFS. The two main questions 
that we need answer are: 
1. Do we need them? what applications use them and how. 
2. Can we have a more simple model that handles just user attributes.

Input is welcome to help me make the case at the meeting.

Thanks, Marc.

From:   Ric Wheeler <[email-masked]>
To:     Dr Fields James Bruce <[email-masked]>, 
Cc:     "Myklebust, Trond" <[email-masked]>, Christoph Anton 
Mitterer <[email-masked]>, Mailing List Linux NFS 
<[email-masked]>, Steve Dickson <[email-masked]>
Date:   10/28/2013 11:32 AM
Subject:        Re: XATTRs in NFS?
Sent by:        [email-masked]

On 10/28/2013 02:08 PM, Dr Fields James Bruce wrote:
> On Mon, Oct 28, 2013 at 02:00:58PM -0400, Ric Wheeler wrote:
>> On 10/28/2013 01:49 PM, Myklebust, Trond wrote:
>>> On Oct 28, 2013, at 12:15 PM, Christoph Anton Mitterer 
<[email-masked]> wrote:
>>>
>>>> On Mon, 2013-10-28 at 11:40 -0400, Ric Wheeler wrote:
>>>>> Then you end up with large directories and an extra name per inode 
that needs to
>>>>> be stored and extra lookups for each file when you do a whole file 
system crawl.
>>>>>
>>>>> Certainly not as easy as adding and xattrs with that information :)
>>>> And I think there's another reason why it wouldn't work...
>>>>
>>>> Imagine I change my system to encode what should be XATTRs in 
hardlink
>>>> pseudo files...
>>>>
>>>> If I have such pair locally e.g. on my ext4:
>>>> /foo/bar/actual/file
>>>> /meta/<SHA512 identifier>.2342348324
>>>>
>>>> And now move/copy the file via the network to the archive, I'd have 
to
>>>> copy both files (which is really annoying), and I'd guess the inode
>>>> coupling would get los (and at least the name wouldn't fit anymore).
>>>>
>>>> So the whole thing is IMHO not even a workaround.
>>> OK. So you're going to do XATTRs for us?
>>>
>>> Trond
>> Now that pNFS is perfect and labeled NFS has made it upstream, I
>> think that Steve D must be looking for something to keep him busy :)
> I agree with Trond that we first really need good evidence about exactly
> who wants this and why.
>
> --b.

For the user space xattrs, many applications store various types of 
metadata. 
Gluster for example heavily uses xattrs, other programs do things like 
data 
scrubbing (look for a long unchanged file, compute a has and store it as 
an 
xattr) or simply use it to annotate the file with the name of the program 
that 
created it. Think of it as file decorations or annotations.

Today, if we store files in NFS that have xattrs, we do in fact cause data 
loss.

I can understand an answer of "this would be hard to do for NFS and need 
to go 
through IETF" but think that xattrs are well enough established in Linux 
and 
supported in the tool chain that it is way too late to question whether or 
not 
supporting them is a worth our time.

Ric

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email-masked]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 28 Oct 2013 20:49:53 +0000

Hi, Marc.

At this point the agenda is full .. based on maximum time.  But I am open to shifting things around a little if the other presenters are willing to adjust their time allotment. 

I suggest that  you and I work this together off the cc'd aliases and you can send a summary when we work out the details.

Spencer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Marc Eshel
> Sent: Monday, October 28, 2013 1:44 PM
> To: [email-masked]
> Cc: Mailing List Linux NFS; Christoph Anton Mitterer; Myklebust, Trond; Steve
> Dickson; [email-masked]; Dr Fields James Bruce; Ric Wheeler; linux-nfs-
> [email-masked]
> Subject: Re: [nfsv4] XATTRs in NFS?
> 
> Hi Spencer,
> Is it still possible to get on the agenda fir IETF 88, I just got approval to travel.
> We can use about 15 minutes, or what ever is available to discussion the
> future of named attributes in NFS. The two main questions that we need
> answer are:
> Mitterer <[email-masked]>, Mailing List Linux NFS <linux-
> [email-masked]>, Steve Dickson <[email-masked]>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Haynes, Tom" <[email-masked]>
Date: Mon, 28 Oct 2013 20:55:06 +0000

> Input is welcome to help me make the case at the meeting.

Be sure to call out why they currently do or do not work.

---

From: "J. Bruce Fields" <[email-masked]>
Date: Mon, 28 Oct 2013 17:02:47 -0400

> Be sure to call out why they currently do or do not work.

And we also need to keep clear the distinctions between:

	- NFSv4 named attributes/streams/forks and "extended attributes"
	  as implemented in Linux and elsewhere.
	- user.* xattrs and "back-door ioctls".

(Your description above seems to confuse named attributes and Linux
extended attributes.)

--b.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 28 Oct 2013 17:04:57 -0400

> (Your description above seems to confuse named attributes and Linux

I wonder if we will have a balanced discussion next week.  The people who seem to know the most about these issues on Linux will not be attending.  Perhaps it would be better to start this discussion on the mailing list instead.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Oct 2013 14:28:34 -0700

"J. Bruce Fields" <[email-masked]> wrote on 10/28/2013 02:02:47 PM:

> From: "J. Bruce Fields" <[email-masked]>
> To: "Haynes, Tom" <[email-masked]>, 
> Cc: Marc Eshel/Almaden/IBM@IBMUS, Spencer Shepler 
> <[email-masked]>, Mailing List Linux NFS <linux-
> [email-masked]>, Christoph Anton Mitterer 
> <[email-masked]>, "Myklebust, Trond" 
> <[email-masked]>, Steve Dickson <[email-masked]>, 
> "[email-masked] [email-masked]" <[email-masked]>, Ric Wheeler 
> <[email-masked]>, "[email-masked]" <linux-
> [email-masked]>
> Date: 10/28/2013 02:02 PM

> (Your description above seems to confuse named attributes and Linux

Yes, I used user which is the Linux type of xattr but what I meant is 
somthing like the Linux user attributes with no special meaning to NFS, so 
NFS can be used to just transfer opaque attributes. 
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Oct 2013 14:28:35 -0700

[email-masked] wrote on 10/28/2013 02:04:57 PM:

> From: Chuck Lever <[email-masked]>
> To: "J. Bruce Fields" <[email-masked]>, 
> Cc: "Haynes, Tom" <[email-masked]>, Marc Eshel/Almaden/
> IBM@IBMUS, Spencer Shepler <[email-masked]>, Mailing List
> Linux NFS <[email-masked]>, Christoph Anton Mitterer 
> Date: 10/28/2013 02:07 PM

> I wonder if we will have a balanced discussion next week.  The 
> people who seem to know the most about these issues on Linux will 
> not be attending.  Perhaps it would be better to start this 
> discussion on the mailing list instead.

Like I said input is welcome to the mailing list and I can try to collect 
all the main points into a presentation at the IETF. The main objective 
for now is to see if this is an item for the working group to start 
dealing with for future NFS protocol enhancements.
Marc.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Oct 2013 15:28:42 -0700

This is a clean start to discuss extended attributes over NFS in the 
mailing list first. 

1. What are the requirements and why? which applications are using them 
and how. 
2. Why are current named attributes in NFS not the right answer.
3. What does the new protocol enhancements should look like? 

Marc.

---

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Oct 2013 15:41:37 -0700

Thanks, Marc.

---

From: Nico Williams <[email-masked]>
Date: Mon, 28 Oct 2013 18:02:29 -0500

> 1. What are the requirements and why? which applications are using them 

Surely the requirements would be semantic compatibility with Linux's
xattrs.  These are defined in manpages such as getxattr(2).  Links can
be easily found online.

: ; man -s2 -k xattr
fgetxattr (2)        - retrieve an extended attribute value
flistxattr (2)       - list extended attribute names
fremovexattr (2)     - remove an extended attribute
fsetxattr (2)        - set an extended attribute value
getxattr (2)         - retrieve an extended attribute value
lgetxattr (2)        - retrieve an extended attribute value
listxattr (2)        - list extended attribute names
llistxattr (2)       - list extended attribute names
lremovexattr (2)     - remove an extended attribute
lsetxattr (2)        - set an extended attribute value
removexattr (2)      - remove an extended attribute
setxattr (2)         - set an extended attribute value
: ; 

Whether there are apps that use them is a different story.  I've no
idea, but here's a github search for code using getxattr:

https://github.com/search?q=getxattr&ref=cmdform&type=Code

Many of the results are from clones of Android, so one has to wade a
fair bit to find apps.  Things like this:

https://github.com/protonet/xattr
https://github.com/jarib/ffi-xattr

make it harder to tell what's using them.

Then we have:

https://github.com/search?q=getxattr&ref=cmdform&type=Issues

which is a bit better, since this brings up more issues with apps using
xattrs and fewer about implementations of xattrs, like:

https://github.com/Cocoanetics/DTFoundation/issues/37

Even then, most of the issues (only 5 pages) relate to filesystems and
FUSE.

Interesting data point: both, an OS X port of ZFS, and ZFSOnLinux felt
the need to add support for xattrs.

The only apps I found this way are the above DTFoundation, Ruby FFI
bindings of xattrs, and:

https://github.com/tobiaswaldvogel/openwrt-addpack
https://github.com/mxcl/homebrew
https://github.com/scrod/nv
https://github.com/ShareKit/ShareKit

> 2. Why are current named attributes in NFS not the right answer.

They have different semantics and namespaces.

xattrs are small, bite-sized.

Named attributes as files within files -"forks"- and align well with
openat(2) but not with *xattr(2).

> 3. What does the new protocol enhancements should look like? 

Probably like direct equivalents of the xattrs system calls as OPs
operating on the current FH.  This seems like the simplest question to
answer.

I'm not advocating adding xattrs to the protocol.  I'm not sure there's
a real case for this.  Developers ought to be using openat(2) more and
xattrs less.  Forks interop better with, e.g., Win32, than xattrs.

Perhaps xattrs could be implemented as a conventional mapping onto named
attributes.  This would require agreement by many people, including
implementors of filesystems that already have both features, so it seems
unlikely, except, perhaps, via reserving named attribute namespace for
implementation purposes, but even that I think is (IIRC) not allowable
for NFSv4 (i.e., the named attribute namespace is already wide open for
use by the public).

This really brings up a different matter: to the extent that xattrs and
named attributes are used to store application data (as opposed to
arbitrary user-named attributes) perhaps we ought to have a registry for
naming them.

Nico
--

---

From: Marc Eshel <[email-masked]>
Date: Mon, 28 Oct 2013 21:27:19 -0700

Hi Trond,
I am just trying to see if there any path here to get xattr over NFS.

Assuming that there is agreement that xattr are used and needed, and we 
need to build a better case here but just give alternative solutions 
doesn't mean that is not needed, almost every thing can be done with an 
additional DB that associates attributes with files but it is not the best 
or right way.

So if we had an NFS protocol extension (might not be named attributes) 
that mapped  to and was semantically compatibly to Linux xattr, as Nico 
wrote, would that be enough to consider adding it to the Linux NFS?

I see the need to show requirement and better interface over the NFS 
protocol (with no security holes)  but not sure if we have another problem 
with standardizing xattr in Linux?

Thanks, Marc. 

From:   "Myklebust, Trond" <[email-masked]>
To:     Christoph Anton Mitterer <[email-masked]>, 
Cc:     Wheeler Ric <[email-masked]>, Anand Avati 
<[email-masked]>, "Dr Fields James Bruce" <[email-masked]>, Mailing 
List Linux NFS <[email-masked]>, Dickson Steve 
<[email-masked]>
Date:   10/28/2013 07:28 PM
Subject:        Re: XATTRs in NFS?
Sent by:        [email-masked]

On Oct 28, 2013, at 9:39 PM, Christoph Anton Mitterer 
<[email-masked]> wrote:

> Trond,... since you ask for motivations pro NFS-XATTRs, I wondered what
> are the strong reasons against?

As I've already told you: you're asking for the addition of a feature 
which is inadequately defined, and is not documented in any standard.

> Of course someone has to do the work,... but that's not really and
> argument pro or con NFS-XATTRs, is it?
> 
> The security problems with namespaces like trusted. or so can probably
> be solved quite easy, e.g. simply by not supporting or
> ignoring/rejecting them.
> 
> You've mentioned caching issues,... could you elaborate a bit on that?
> How is that much different from caching any other file metadata NFS
> transfers?

The standard metadata such as ctime, mtime, size, .... are all part of an 
existing NFS caching model called close-to-open. We know how they change 
when the filesystem data changes.

How do I know when it is safe to cache your checksum xattr? I don't even 
know what it is, let alone what its relation is to the other filesystem 
objects.
Let's say client B modifies the file and updates the checksum. When client 
A opens the file, it will see that the data has changed, but how does it 
know that it also needs to update the xattr information?
Alternatively, let's say that client A reads the file and checksum data 
after client B has updated the file, but before it updates the checksum. 
What will cause client A to stop caching the stale checksum once client B 
has updated it?

Trond--

---

From: Anand Avati <[email-masked]>
Date: Thu, 31 Oct 2013 13:54:47 -0700

> -------- Original Message --------
> Subject:     XATTRs in NFS
> Date:     Mon, 28 Oct 2013 15:41:37 -0700
> From:     Marc Eshel <[email-masked]>
> To:     Marc Eshel <[email-masked]>
> CC:     Mailing List Linux NFS <[email-masked]>,
> [email-masked], [email-masked]
>
>
>

Reiterating, one of the applications (glusterfs - a distributed 
filesystem) has been using filesystems with extended attribute support 
by annotating files and dirs with crucial info for its operation (uses 
xattrs as a dumb key/value store). Usage of xattrs is crucial for the 
operation as glusterfs avoids using any sort of "external" meta-data 
database or metadata server for its operation (as a philosophy/design 
principle focused on robustness). Lack of xattrs on NFS prevents using 
NFS volumes as gluster bricks.

To that end, if NFS can map user.XXX values to user.XXX named attributes 
and vice versa in the NFS client and NFS server respectively, that would 
make NFS volumes be usable as gluster bricks.

> 2. Why are current named attributes in NFS not the right answer.

I think they might be, if we can map user.XXXX xattrs to named 
attributes and avoid any sort of interpretation of they key/values by 
NFS. Though from my point of view, all I care is for getxattr/setxattr 
of user.XXX keys to "just work" (whether they are mapped to named 
attributes or implemented as a separate feature).

Avati

---

From: Nico Williams <[email-masked]>
Date: Thu, 31 Oct 2013 16:36:33 -0500

On Thu, Oct 31, 2013 at 01:54:47PM -0700, Anand Avati wrote:
> attributes and avoid any sort of interpretation of they key/values
> by NFS. Though from my point of view, all I care is for
> getxattr/setxattr of user.XXX keys to "just work" (whether they are
> mapped to named attributes or implemented as a separate feature).

That's an open question.  Are writes to xattrs supposed to be atomic?

Anyways, it seems at least one filesystem (XFS) maps xattrs onto a
hidden data fork (named attribute) for that purpose.  So, I'm guessing:
a) no, they're not atomic, b) named attributes should work... c) if
only we had a namespace to steal names from for this purpose.

Nico
--