Subject: [nfsv4] named attributes vs. xattrs pre-meeting

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 5 Nov 2013 09:35:23 -0500 (EST)

Hi NFSv4,

Per Spence, if time, permits, Marc Eshel and I may (or may not) have a few minutes at the
end of the meeting to direct attention to the recently (re)opened named attributes vs.
Linux (etc) extended attributes debate.

Here are some pre-meeting notes:

1. Motivation (rehash)

RFC5661 (and RFC3530) provides a mechanism ('named attributes') for associating arbitrary,
named specific data with files and directories.  The NFSv4 mechanism resembles file forks
or alternate data streams, in that data is seekable and of arbitrary size.

Unix-like platforms (Linux, FreeBSD, OpenBSD, Darwin) provide a family of "extended attribute"
mechanisms (xattrs) to associate named, specific data with files.  Xattrs use a
listkeys/getbykey/setbykey interface, in which get and set are atomic, so presumed to
be of limited size--but xattr size limits are (filesystem) implementation defined,
apparently ranging up to 64K for some filesystems.
 
There now appears to be relatively strong interest in the Linux community in exposing
xattrs over NFSv4 by -some- means (though the matter is not settled).  Nevertheless, the obvious
mechanism for exposing xttrs to NFSv4--mapping xattrs onto NFSv4 named attributes--has been
claimed to be impossible to do correctly or safely.

2. Prehistory.

There is a standing debate whether xattrs may be mapped onto named attributes,
as presumably the RFC5661 authors believed possible, but several implementers have disputed.
The prior WG debate on the topic is available in the following email thread:

 http://www.ietf.org/mail-archive/web/nfsv4/current/msg02360.html

3. At Issue in 2013

In 2005, there was a minority viewpoint in the WG, represented by Gordon Waidhofer, that a
mapping from xattrs to named attributes was ill advised, specifically because Waidhofer
believed Linux and other Unix-like platforms would soon (by 2007) support stream attribute/subfile
interfaces natively, and that these rather than "OS/2-like," "weenie" xattr values should
be the ones exposed over NFSv4. As an alternative, interim standardization of specific
attributes one-by-one as IETF drafts was proposed.

In the Linux environment of 2013, "weenie" attributes appear to be on the ascendant, and a
significant number of participants in the Linux file system community seeks to expose xattrs
over NFSv4, in some manner.

The question we find relevant to this WG is whether NFSv4 named attributes as currently standardized
can, or, possibly, should be used to represent xattrs, and, in either case, what language, adjustment,
or alternate protocol mechanism(s) should be proposed.

4. xattrs "subtly incompatible" with xattrs (Chuck)

Drawing on the old discussion, there seem to be at least the following arguments:

a. xattrs are of finite size, streams unbounded (Robert Watson, FreeBSD)

A hypothetical Linux server implementation which maps named attributes to local xattrs will
be unable to store stream attributes, or worse, fail to store them when they exceed an
(undefined) size.

So, this is a claim that a non-Linux or future Linux client application might be harmed when
interacting with a Linux implementation over NFSv4, which it incorrectly assumes can store stream
attributes.

b. xattrs get and set operations are atomic, while named attributes are not.

As argued here:

http://www.ietf.org/mail-archive/web/nfsv4/current/msg02356.html

the problem of atomicity appears potentially fixable with some convention regarding
reads/writes to named attribute files, but some may have found the argument
uncompelling.

c. Caching

Apparently, a named attribute model which provides for get/set atomicity is in conflict with
any idea of attribute caching.

The simple form of the argument that xattr interfaces conflict with current NFSv4 consistency
semantics (close-to-open [Trond 2013], delegation [mentioned by Brent Callaghan, 2005], can likely be
disposed of easily [Nico, 2005].  There may be room for additional language in the standard
binding the server, and requesting the client, not to kill kittens.

d. Local system namespace mapping issues.

Linux has given string prefixes "system.*" "trusted.*" "user.*" special meanings, 
in particular "system.*" apparently could not be safely exposed to remote systems,
or without special constraints.  On Linux, only "user.*" attributes are really
discretionary.

It appears well established that "user xattrs are in principle harmless" (Trond, 2013),
ie, to expose over NFS using existing permission models.  Apparently platforms must
define a set of rules indicating which attributes could be exposed.  On Linux, this might
include the "user.*" prefix namespace.

5. Other Potential Issues.

a. It has been claimed that NFSv4 lacks a way to atomically apply named attributes to a file
on create.  A requirement for such behavior is easy to motivate (for example, David Quigley
initially found a requirement for this in Labeled NFS).

b. Claimed problems with the NFSv4 named attribute model per se.

NFSv4 named attributes re-use NFSv4 filehandle and directory objects, with restrictions, to
provide enumeration of named attributes and perform attribute i/o.  Engineers at CITI found
that student implementers had difficulty adding named attribute support to Linux NFS, due to
interface issues.  On the other hand, at least two clients (Solaris, Windows) have implemented
named attributes without issue.  We (Cohortfs) are attempting to implement the interface
in the Linux client, to better understand the issues.

5. Action Items for NFSv4 WG

We believe there is demonstrably sufficient motivation for the NFSv4 WG to provide a
mechanism suitable for exposing xattrs over NFSv4.

We would propose that initial effort should focus on whether the named attribute interfaces
already standardized could be adjusted or clarified so as to provide this support without
additional work, but alternatively, to standardize new interfaces, and well-define the
relationship between the new interfaces and those already standardized.

Matt and Marc

-- 
Matt Benjamin
The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 09:46:16 -0600

> b. xattrs get and set operations are atomic, while named attributes are not.

I don't believe this.  I looked at *one* implementation, the one for
XFS, in Linux, and it maps xattrs onto a hidden named attribute file
whose contents is basically an encoding of all the xattrs of the file.
This seems like somewhat of a precedent setter.  Perhaps a canvassing of
what *other* filesystems due is in order.

I might be wrong but I saw nothing in XFS to ensure atomicity, except,
perhaps, if the named attribute in question is treated as an append log,
then atomicity would be possible.  And *that*'s one way that I'd
implement atomicity (if I had to) in an xattrs-over-named-attributes
design (see below).

The Linux *xattr(2) manpages say nothing about atomicity either.

The fact that the *xattr(2) system calls read and write the entire xattr
at once does not imply atomicity: it only makes it possible.

> reads/writes to named attribute files, but some may have found the argument

Some existing filesystems already do store xattrs in named attributes.
I find that a compelling argument for doing the same in NFSv4.

The only real problems are: a) the need to agree on all details of the
mapping to named attributes, b) named attr namespace issues.

BTW, I can see at least two ways to get atomicity in an
xattrs-over-named-attrs implementation:

1) One named attr per-xattr, rename temp named attrs it into place
   (rename is atomic).

2) Use a single named attr as a garbage-collected copy(append)-on-write
   log of xattrs, using byte range locking to synchronize reads and
   writes.

Obviously there's more.  Lots of apps manage just fine to do things like
this using normal POSIX interfaces.

> any idea of attribute caching.

See above.  The same arguments as to how to implement atomicity (if
required) -namely that this is like any other POSIX (and Win32)
application- applies to caching.

> Linux has given string prefixes "system.*" "trusted.*" "user.*"
> special meanings, in particular "system.*" apparently could not be
> safely exposed to remote systems, or without special constraints.  On
> Linux, only "user.*" attributes are really discretionary.

Yesh, namespace issues come up.  I think we need a way to reserve named
attribute sub-namespaces anyways.  XFS uses a hidden named attribute to
store xattrs; we could copy that in NFSv4, but I think it'd be cleaner
to not create a new namespace, just carve out a part of the existing
one, with a registry of sub-namespace prefixes.

> It appears well established that "user xattrs are in principle
> harmless" (Trond, 2013), ie, to expose over NFS using existing
> permission models.  Apparently platforms must define a set of rules
> indicating which attributes could be exposed.  On Linux, this might

Actually, we might need new ACE bitmask bits for xattrs, if we don't
want to depend on ACE4_{READ,WRITE}_ATTRIBUTES.  This is orthogonal to
the namespace issues.

> a. It has been claimed that NFSv4 lacks a way to atomically apply
> named attributes to a file on create.  A requirement for such behavior
> is easy to motivate (for example, David Quigley

Is there a way to do this on Linux, FreeBSD, ...?  I don't see it in
Linux.

> NFSv4 named attributes re-use NFSv4 filehandle and directory objects,
> with restrictions, to provide enumeration of named attributes and
> perform attribute i/o.  Engineers at CITI found that student
> implementers had difficulty adding named attribute support to Linux
> NFS, due to interface issues.  On the other hand, at least two clients
> (Solaris, Windows) have implemented named attributes without issue.

It's very difficult to believe the report of difficulty "due to
interface issues" on Linux, if by "interfaces" you mean openat(2) and
friends, considering that Solaris and others managed.

> We (Cohortfs) are attempting to implement the interface in the Linux
> client, to better understand the issues.

Excellent.

> We believe there is demonstrably sufficient motivation for the NFSv4
> WG to provide a mechanism suitable for exposing xattrs over NFSv4.

I agree.

> We would propose that initial effort should focus on whether the named
> attribute interfaces already standardized could be adjusted or
> clarified so as to provide this support without additional work, but
> alternatively, to standardize new interfaces, and well-define the

I agree.  Start with a mapping of xattrs onto named attributes.  If that
turns out to be infeasible, then look at adding first-class xattrs to
the protocol.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 09:53:39 -0600

On Tue, Nov 05, 2013 at 09:46:16AM -0600, Nico Williams wrote:
> I don't believe this.  [...]

Also, and to repeat myself in case I was too blunt, named attributes are
no more and no less atomic than normal files.  Which is to say that the
full power of POSIX and NFSv4 is available to make writes to named
attributes atomic (rename into place, byte range locking +
copy-on-write).  Atomicity of xattrs is a non-issue.

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 10:09:50 -0600

Can you do an experiment?  Here's my proposal:

   Write an LD_PRELOAD interposer on *xattr(2) whose *xattr(2)
   interposers do this:

    - call the real thing (dlsym(RTLD_NEXT, ...))
    - if the real thing fails with errno == ENOTSUP then call an
      implementation that uses named attributes (openat(2))

   The implementation that uses named attributes should do something
   like this for writing xattrs:

    - mkstemp() .xattr--XXXXX as a named attribute
    - write the xattr data into the temp named attribte
    - renameat() the temp named attr into place as .xattr-<xattr-name>

   while for reading openat() the .xattr-<xattr-name>, and for listing
   readdir the named attributes and look for .xattr-<xattr-name>.

   Then test an xattrs app with this LD_PRELOAD shim using an NFSv4
   filesystem, on an NFSv4 client.

If that works then all we need to do standards-wise is:

 - carve out namespace for a mapping of xattrs onto named attributes

 - specify in excruciating detail the mapping onto named attributes

Heck, you might even contribute the interposer to glibc as the new
xattrs syscall stubs.

One issue is that we might need to coordinate the namespace carve out
with the Open Group.  That might argue for creating a hidden named
attributes namespace so we don't have to coordinate with the Open Group.
However I think it'd be best to have a registry anyways.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 10:24:47 -0600

On Tue, Nov 05, 2013 at 08:19:10AM -0800, Christoph Hellwig wrote:
> On Tue, Nov 05, 2013 at 10:09:50AM -0600, Nico Williams wrote:
> Why the heck would glibc accept that if we have a perfectly working
> solution that is way more elegant than the filestreams hack?
> 
> With my Linux VFS developer hat on I can also guarantee you that we will
> not put in the slightest amount of support for the ill-advised NFSv4
> subfiles in. 

You wouldn't have to.  Linux already supports openat(2) and friends.
This is not in the VFS layer.

As for "ill-advised", please substantiate that with reasoned arguments.
I did for my proposal.  I also don't mind if something else is done; I'm
not wedded to any one thing.  My proposal is based on doing the least
violence to NFSv4 while also making it possible to implement xattrs as
much as possible in user-land on systems that don't have xattrs but do
have openat(2) and friends -- I think that's a reasonable motivation, or
if not, I'd love to hear why not.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 10:43:07 -0600

On Tue, Nov 05, 2013 at 08:31:48AM -0800, Christoph Hellwig wrote:
> On Tue, Nov 05, 2013 at 10:24:47AM -0600, Nico Williams wrote:
> Linux does support openat, but not O_XATTR.  As you might know the *at
> syscalls primary purpose is to avoid relying on the current working
> directory, and not the subfiles that are grafted onto it and causes
> various semantic problems like for file locking.

I see.  This doesn't preclude mapping xattrs onto named attrs, it just
means that the NFSv4 client proper would have to do it, that it couldn't
be left to user-land.

> The IRIX/Linux/BSD style attributes are a simple way to attach
> name-value pairs to an inode, without dragging along the garbage of a
> file Posix inode or file state for access to it.  Note that I'm not

This isn't the vger kernel list, FYI.

> nessecarily arguing for adding them to NFSv4 here, as their prime usage
> is for implementing other functionality inside the filesystem.

But people want NFSv4 to handle xattrs because applications rely on
them to be persistent and visible to other clients.

> I do however argue for a deprecation of the subfiles - a protocol should
> specify what the majority of the client implementation can make use of,
> something NFSv4.0 failed on badly both with the subfiles and the ACLs.

Noted.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Tue, 5 Nov 2013 09:00:28 -0800

>> WG to provide a mechanism suitable for exposing xattrs over NFSv4.

Although Matt's e-mail is detailed, we (the IETF) still do not have any use cases.  It's difficult to design something when you don't know how it will be used.  It's difficult for implementers to test something that has no use cases to verify.  It's difficult for maintainers to know whether it is worth their time to support an implementation for the long term.

> turns out to be infeasible, then look at adding first-class xattrs to

I don't see a hard ordering.  You can and should try both now.

In the abstract, I favor a first-class xattr implementation.  That way a file server that supports only xattrs would not have to expose a weak named attribute implementation to those client implementations that expect a full named attribute implementation.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 11:38:28 -0600

On Tue, Nov 05, 2013 at 09:00:28AM -0800, Chuck Lever wrote:
> Although Matt's e-mail is detailed, we (the IETF) still do not have
> any use cases.  It's difficult to design something when you don't know
> how it will be used.  It's difficult for implementers to test
> something that has no use cases to verify.  It's difficult for
> maintainers to know whether it is worth their time to support an
> implementation for the long term.

This might be useful:

http://gentoobrowse.randomdan.homeip.net/use/xattr

Use-cases for xattrs will generally be hard to come by because apps
developers don't have to come to us to tell us that they're using them.
I find lots of FFIs for xattrs (for Ruby, Haskell, Python, ...); they
must exist for a reason.

It seems Spotlight uses them too:

http://chapados.org/2007/11/04/leopard-shines-the-spotlight-on-xattrs
https://getsatisfaction.com/binaryage/topics/use_extended_attributes_xattr_on_webdav_and_smb_shares_to_avoid_file_pollution

Related:

http://ironicsoftware.com/vanillaforums/discussion/755/developer-thoughts-on-adopting-openmeta/p1

This is nice:

http://curl.haxx.se/mail/archive-2010-11/0009.html

Oh, are Linux file capabilities stored in xattrs:

http://adsm.org/forum/showthread.php?27938-Linux-file-xattrs

?

Something interesting regarding namespaces:

http://superuser.com/questions/61185/why-do-i-get-files-like-foo-in-my-tar

> In the abstract, I favor a first-class xattr implementation.  That way
> a file server that supports only xattrs would not have to expose a
> weak named attribute implementation to those client implementations
> that expect a full named attribute implementation.

That's a good argument for first-class xattrs.  Would we want to allow
server implementors to map them onto named attributes?  I think so.  And
I think both, named and xattrs need a namespace registry given that
system apps (e.g., backup tools) use xattrs.

Nico
--

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 5 Nov 2013 12:43:53 -0500 (EST)

Hi,

I found out last evening that IBM has a number of use cases, most detailed, IBM
research believes that xattrs (with atomic application of xattrs to newly
created objects) are a requirement to one or more OpenStack object protocols on
NFSv4.

Matt

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 5 Nov 2013 17:51:46 +0000

Does the "xattr" need to be updated atomically or would a mechanism used by the application/client to verify the data has not been "torn" on the write.  IMO an atomic is a non-starter.  Using techniques like checksums of the data or the like would be a better method given the tools we have in the underlying filesystems.

Spencer

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Matt W. Benjamin
> Sent: Tuesday, November 5, 2013 9:44 AM
> To: Nico Williams
> Cc: nfsv4
> Subject: Re: [nfsv4] named attributes vs. xattrs pre-meeting
> 
> Hi,
> 
> I found out last evening that IBM has a number of use cases, most detailed,
> IBM research believes that xattrs (with atomic application of xattrs to newly
> created objects) are a requirement to one or more OpenStack object
> protocols on NFSv4.
> 
> Matt
> 
> > https://getsatisfaction.com/binaryage/topics/use_extended_attributes_x
> > attr_on_webdav_and_smb_shares_to_avoid_file_pollution
> >
> > Related:
> >
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 5 Nov 2013 13:00:02 -0500 (EST)

While atomic update for a large byte sequence seems impossible, I'm not
clear why we could not ensure what xattr (thinks it) is relying on
for a finite sequence which could be sent in a single write/compound?

Matt

----- "Spencer Shepler" <[email-masked]> wrote:

> Does the "xattr" need to be updated atomically or would a mechanism
> used by the application/client to verify the data has not been "torn"
> on the write.  IMO an atomic is a non-starter.  Using techniques like
> checksums of the data or the like would be a better method given the
> tools we have in the underlying filesystems.
> 
> Spencer
> 
> > > > On Nov 5, 2013, at 7:46 AM, Nico Williams
> <[email-masked]>
> > > wrote:
> > > > >> We believe there is demonstrably sufficient motivation for
> the
> > > NFSv4
> > > > any use cases.  It's difficult to design something when you
> don't
> > > know
> > > > > I agree.  Start with a mapping of xattrs onto named
> attributes.
> > > If that
> > > > > turns out to be infeasible, then look at adding first-class
> xattrs
> > > to
> > > > > the protocol.
> > > >
> > > > In the abstract, I favor a first-class xattr implementation. 
> That
> > > way
> > > server implementors to map them onto named attributes?  I think
> so.
> > > And

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 12:18:25 -0600

> for a finite sequence which could be sent in a single write/compound?

xattrs have a small max size, which should make atomicity OK for an
NFSv4 xattrs extension (since NFSv4 can handle a number of atomic ops,
like exclusive creates and renames).

It's not clear that xattrs are expected to be atomic, but this sort of
thing:

http://www.mail-archive.com/[email-masked]/msg08785.html

makes me think that for small enough xattrs they are expected to be
atomic.

(I've no idea about write visibility (caching) semantics of xattrs.  But
it's a safe bet that xattr writes are expected to be visible to all
subsequent readers.  A naive approach would be to not cache them at all
and see who screams about performance.  One of the nice things about
using named attributes for xattrs is that caching is addressed
indirectly.)

Nico
--

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 5 Nov 2013 18:20:34 +0000

> From: Nico Williams [mailto:[email-masked]]
> Sent: Tuesday, November 5, 2013 10:18 AM
> To: Matt W. Benjamin
> Cc: Spencer Shepler; nfsv4
> > > on the write.  IMO an atomic is a non-starter.  Using techniques
> > > like checksums of the data or the like would be a better method
> > > given the tools we have in the underlying filesystems.
> >
> > While atomic update for a large byte sequence seems impossible, I'm
> > not clear why we could not ensure what xattr (thinks it) is relying on
> NFSv4 xattrs extension (since NFSv4 can handle a number of atomic ops, like
> exclusive creates and renames).
> 
> makes me think that for small enough xattrs they are expected to be atomic.
> 
> (I've no idea about write visibility (caching) semantics of xattrs.  But it's a safe
> bet that xattr writes are expected to be visible to all subsequent readers.  A
> naive approach would be to not cache them at all and see who screams
> about performance.  One of the nice things about using named attributes for
> xattrs is that caching is addressed
> indirectly.)

[Spencer] My concern about atomic is that we don't have any requirement in the protocol at the moment about atomicity and relying on xattr atomicity seems too implementation specific.

Spencer

---

From: "Matt W. Benjamin" <[email-masked]>
Date: Tue, 5 Nov 2013 13:30:11 -0500 (EST)

I think perhaps use of the word SHOULD might be helpful here.  A hypothetical Linux
implementation mapping attributes to xattrs shouldn't need to provide stronger
atomicity guarantees than its underlying filesystem.

Matt

----- "Spencer Shepler" <[email-masked]> wrote:

> [Spencer] My concern about atomic is that we don't have any
> requirement in the protocol at the moment about atomicity and relying
> on xattr atomicity seems too implementation specific.
> 
> Spencer

http://linuxbox.com

tel.  734-761-4689 
fax.  734-769-8938 
cel.  734-216-5309

---

From: Robert Thurlow <[email-masked]>
Date: Tue, 05 Nov 2013 11:48:24 -0700

> claimed to be impossible to do correctly or safely.

I have no doubt we could write guidance about how a client should
use the named attribute mechanism to act like xattrs - we could
carve out some namespace, specify locking, etc.  But I wonder
if that's the right idea.  A client application trying to use
xattrs with a server that supported xattrs directly would get
a lower level of service, and could fail to cooperate on a local
application running on the server unless a shim layer was also
implemented on the server side ("if it's a file named .xattr*,
ignore the lock/unlock traffic and just use VOP_XATTR()").

I know it's hard to wait for the standard to be revised, but
would it make more sense to define new, atomic key-value type
attributes in the next revision of the protocol and let a server
implementor map that onto their local named attributes if that
was their best option?  What would be the reasons to refuse to
add protocol support?

Rob T

---

From: Nico Williams <[email-masked]>
Date: Tue, 5 Nov 2013 14:32:56 -0600

> ignore the lock/unlock traffic and just use VOP_XATTR()").

Yes, this is another good argument for first-class xattrs.  There's now
two convincing (for me) arguments for first-class xattrs.  My earlier
support for mapping xattrs onto named attributes is withdrawn.

Nico
--

---

From: Chuck Lever <[email-masked]>
Date: Wed, 6 Nov 2013 08:46:17 -0800

> makes me think that for small enough xattrs they are expected to be

I think we're considering only user. attributes.  This kind of security use case is out of scope, and already covered by labeled NFS anyway.  Are there other uses that might need "atomicity" ?

I'm not really clear on what people mean when they mention an atomicity requirement.  Are we talking about the ability to write the contents of one xattr in one operation?  The ability to update all of a file's xattrs at once?  A guarantee that the xattr contents will never suffer from a torn write on the server?  That the server does not expose partially written xattr contents to other clients?

> using named attributes for xattrs is that caching is addressed

We might go a step further and protect xattr contents with delegation.  In other words, if a server provides a READ delegation to a client, the server promises to call the client back if another client wants to modify one of the file's xattrs.

---

From: Nico Williams <[email-masked]>
Date: Wed, 6 Nov 2013 11:11:41 -0600

On Wed, Nov 06, 2013 at 08:46:17AM -0800, Chuck Lever wrote:
> I think we're considering only user. attributes.  This kind of
> security use case is out of scope, and already covered by labeled NFS
> anyway.  Are there other uses that might need "atomicity" ?

Backup tools might record backup times and such.  I think they'd need
atomicity as well.

> I'm not really clear on what people mean when they mention an
> atomicity requirement.  Are we talking about the ability to write the
> contents of one xattr in one operation?  The ability to update all of
> a file's xattrs at once?  A guarantee that the xattr contents will
> never suffer from a torn write on the server?  That the server does
> not expose partially written xattr contents to other clients?

% man -s2 -k xattr
fgetxattr (2)        - retrieve an extended attribute value
flistxattr (2)       - list extended attribute names
fremovexattr (2)     - remove an extended attribute
fsetxattr (2)        - set an extended attribute value
getxattr (2)         - retrieve an extended attribute value
lgetxattr (2)        - retrieve an extended attribute value
listxattr (2)        - list extended attribute names
llistxattr (2)       - list extended attribute names
lremovexattr (2)     - remove an extended attribute
lsetxattr (2)        - set an extended attribute value
removexattr (2)      - remove an extended attribute
setxattr (2)         - set an extended attribute value
% 

The API does not allow random I/O within xattrs.  It deals in whole
xattrs at all times, and the size of xattrs is limited.  Each system
call -other than the listing ones- deals in a single xattr and there's
no transactional composition.

Therefore the API *clearly* does NOT provide for multi-xattr atomic
updates.

A more thorough survey of the docs for all the OSes that have "xattrs"
is probably in order, just in case.  My impression is that all of them
are as described here.

The API allows/requires atomicity as follows:

 - Each setxattr operation's new xattr value is visible as provided by
   the client app (xattr name, contents, and length), or not at all.

   No partial success, at least for xattr values smaller than some max
   size.

   This is neither stated nor implied, but surely anything less than
   this would make xattrs dangerous for many uses, so I think we should
   assume atomiticy in this sense is required.

 - Racing XATTR_REPLACE clients will not produce garbage.

   Also neither stated nor implied, but surely desirable, and would
   follow from the above.

 - XATTR_CREATE is like O_EXCL | O_CREATE, so of N clients racing to
   XATTR_CREATE only one will win.

   This one is required.

> > using named attributes for xattrs is that caching is addressed

I should say that anything less than POSIX-like write visibility would
be surprising anyways.  There's no byte range locking to mediate
visibility, but if we assume atomicity then that's enough: reads
following an XATTR_CREATE or XATTR_REPLACE must see it.

> We might go a step further and protect xattr contents with delegation.
> In other words, if a server provides a READ delegation to a client,
> the server promises to call the client back if another client wants to
> modify one of the file's xattrs.

A delegation for all xattrs would work, and so would a delegation
per-xattr.  Considering how small xattrs are expected to be, ISTM that a
delegation per-xattr is overkill and will use up too many resources.

Nico
--

---

From: "Myklebust, Trond" <[email-masked]>
Date: Wed, 6 Nov 2013 17:45:09 +0000

> I'm not really clear on what people mean when they mention an atomicity requirement.  Are we talking about the ability to write the contents of one xattr in one operation?  The ability to update al…

The Linux setxattr operation will create the xattr atomically if you supply the XATTR_CREATE flag.
i.e. if another process tries to read using getxattr(), it will either receive an ENOATTR (the xattr does not exist), or it will receive an ERANGE (the buffer is too small), or the buffer will be filled with an exact copy of the buffer argument to the setxattr() call.

If you supply the XATTR_REPLACE flag, then the xattr contents will be replaced atomically.
i.e. a getxattr() call will receive an ERANGE (the buffer is too small), or the buffer will be filled with either an exact copy of the contents of the xattr before the setxattr() call, or an exact copy of the contents of the buffer argument to the setxattr() call.

While there is no standard for xattr behaviour, I believe that ordinary MacOS user xattrs are expected to behave the same way. Apparently resource forks (== xattr with the magic name “com.apple.ResourceFork") are a special case - they appear to be closer to named attributes.

> We might go a step further and protect xattr contents with delegation.  In other words, if a server provides a READ delegation to a client, the server promises to call the client back if another cl…

Extending delegations to cover user xattrs would fit that model, and makes sense, since there is no other natural caching mechanism available: close-to-open does not apply if changes and reads are atomic.

  Trond