Subject: RE: fsync() fails under NFS, right?

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 13 Sep 1999 09:43:11 -0700

> 
>   o Several questions --
>   o 
>   o When you mentioned "superblock" are you refering to block 
> on a UFS file
>   o system which contains the file system configuration 
> information and
>   o such?  If so, fsync(2) has nothing to do with that.
> 
> I used an overloaded term, mea culpa.  The
> superblock I'm talking about has nothing to do
> with an underlying system.  It's a part of the
> user file implementing our transactional
> database.  A "commit" is done by writing a certain
> sector-size block atomically into the file.  For
> mental health of all those involved, it is vital
> that the fact of such write-through is known for sure.

If you truly mean "atomically", then fsync working
properly is not sufficient.  The semantics of fsync
provide that by the time fsync returns the write of 
that "sector-size block" has been committed to disk
(or at least to some form of stable storage) but you
don't have any guarantee (within v3 or v4 as currently
specified) that the write takes effect atomically.
Most servers will do the write atomically given a 
reasonably-sized (and aligned) write, but the protocol
does not require it.  The server may compress files
so what you think is properly aligned may not be on
his disk, etc.

I don't see anything in the spec that says that a
WRITE has to be atomic in the sense that a READ cannot
see an in-progress partially-completed WRITE.  You can 
use file locking if you need that kind of guarantee.

The big problem is atomicity in the face of failures.
Locking doesn't help you here.  As far the protocol
goes, there is no guarantee that an unacknowledged
WRITE is either completely-done or not-done-at-all
after a server failure.  Many servers have the
property that this is true for some range of write
sizes and alignments and some servers can provide it
for all writes but it is not a protocol reqruirement
as far as I know.

Making that a protocol requirement would be too onerous
in general.  I think it would be good idea to give the
server a way to inform the client of what atomicity
guarantees he provides.  A per-fsid attribute would
be nice, if we can figure out what would go in it.

---

From: Gordon Waidhofer <[email-masked]>
Date: Mon, 13 Sep 1999 10:14:53 -0700

Noveck, Dave wrote:

> > ...  A "commit" is done by writing a certain

I'm going to make an educated guess here that these folks
are using a bottom-up tree substitution technique.
(Log Structure file system is a good example of a
variant of this technique). The "superblock" is a
root of the tree.

The sequence they rely upon is:

   WRITE(s) of leafs and interior nodes
   FSYNC
   WRITE (exactly one) of new root
   FSYNC

The atomicity of the multiple updates is provided by
the ordering of the write-through writes. If the
first write()/fsync() is incomplete before the
second write()/fsync() has effect on the disk
image, their atomicity story crumbles. This issue
exists whether NFS or local file system. The original
posting simply asked whether fsync() works properly
over NFS. Peter S. rightly said it depends more on
the client than anything. Eric W. asked for a
reproducible case so that we could all look at
the problem, which might actually be the test case.
Nevertheless, the expectation that fsync() work
properly (not return until the deed is truly done)
is reasonable.

> use file locking if you need that kind of guarantee.

True. Though the original poster did not raise this
question, so I'm taking it for granted that they have
exclusion under control. Again, this is not peculiar to NFS.

> for all writes but it is not a protocol reqruirement

This issue exists whether NFS, UFS, or straight disk.
It's the damaged sector problem. I'll bet that the
design the original poster is working with already
takes this into account. If a "root" (superblock)
is flawed, the previous root is used.

> be nice, if we can figure out what would go in it.

Just an extended attribute(s) on the actual file would
be fine. Some file services may have special provisions
in their file model for such a requirement. As such,
the attribute would not be fsid wide, but rather
file specific.

Again, I don't think the poster is asking for the protocol
to support transactions, ala Seltzer et al. Rather,
just that the normal POSIX/UNIX semantics for fsync()
be dependable over NFS. As such, all other issues
of ordering, fault tolerance, and exclusion are likely
addressed by their design. Just an educated guess, though.

-- 
==================================================================
  Gordon W Waidhofer                    Traakan Software, Inc.
  [email-masked]                       111 Main St., Suite #5
  650-947-1583  Direct                  Los Altos, CA  94042
  650-947-1580  Main                    http://www.traakan.com
  650-947-1588  Fax