Subject: Re: fsync() fails under NFS, right?

From: Peter Staubach <[email-masked]>
Date: Fri, 10 Sep 1999 11:34:00 -0700 (PDT)

This doesn't really have anything current to do with NFS Version 4,
but I thought that I would respond anyway.

> 
> Greetings, brave NFS trailblazers!
> 
> I have to ask you for a definitive confirmation or
> refutation of the following sad observation:
> 
> fsync() (2) fails under NFS.
> 
> Why ask: I implemented a simple and robust
> transactional layer abstraction, called bedrock,
> for the NECI Intermemory project -- see
> www.intermemory.org.  We currently finish the
> paper (perhaps for USENIX), and the code will be
> GPL'ed on the web.  
> 
> As a part of the bedrock spec, we had to disallow
> remote files as the bedrock files (we currently
> rely on files, raw partitions will come later...
> if ever.:-)  
> 
> The transactional _commit_ is traditionally
> implemented as "sliding in" a _superblock_ once
> the rest is ready for enabling.  It is vital that
> the commit reports success ONLY when the
> superblock write actually ended up on disk.  On
> UNIX, fsync() is supposed to do just that.
> 
> However, if the file if NFS mounted, fsync LIES.
> :-)  It reports success while the file was just
> too huge to be written to any, even Martian,
> disk.  Why can't NFS make fsync() report an
> "undefined" status instead of lying blatantly?
> 
> Other fine points discovered while battling the
> lying NFS fsync(): detecting a remote system
> status is difficult under unix/Linux.  I use GNU
> fileutils now, which are idiosyncratic, heavily
> system-specific, etc.
> 
> Overall, it is vital for transactional processing
> to be able to detect whether a file system
> supports fsync(), and so the new NFS must have
> fsync() operate correctly or have a way to report
> it's unreliable.
> 
> Am I right in any of the above?  ;-)
> 

Several questions --

When you mentioned "superblock" are you refering to block on a UFS file
system which contains the file system configuration information and
such?  If so, fsync(2) has nothing to do with that.

The NFS protocol(s) don't know anything about fsync(2).  It is up the
individual client implementation to support fsync(2) or not, depending
upon its own set of semantics.  If your particular client doesn't
support fsync(2) correctly, then I would suggest complaining to the
vendor which supplied your NFS implementation.  It is just a bug.
So, what is the platform which you are using?

	Thanx...

		ps

---

From: [email-masked]
Date: Fri, 10 Sep 1999 17:16:48 -0400

o This doesn't really have anything current to do with NFS Version 4,
  o but I thought that I would respond anyway.

Well, I just discovered the list, but you guys
discussed transactional enhancements to NFS.  If
you don't know whether something is on disk or
not, how can any such transactionality be
ensured?!  My question is based on observing NFS
behavior on Sun and Linux.  On both, fsync() is
misleading, using a presidential adjective.  :-)

  o Several questions --
  o 
  o When you mentioned "superblock" are you refering to block on a UFS file
  o system which contains the file system configuration information and
  o such?  If so, fsync(2) has nothing to do with that.

I used an overloaded term, mea culpa.  The
superblock I'm talking about has nothing to do
with an underlying system.  It's a part of the
user file implementing our transactional
database.  A "commit" is done by writing a certain
sector-size block atomically into the file.  For
mental health of all those involved, it is vital
that the fact of such write-through is known for sure.

  o The NFS protocol(s) don't know anything about fsync(2).  It is up the
  o individual client implementation to support fsync(2) or not, depending
  o upon its own set of semantics.  If your particular client doesn't
  o support fsync(2) correctly, then I would suggest complaining to the
  o vendor which supplied your NFS implementation.  It is just a bug.
  o So, what is the platform which you are using?

Linux and Sun both return success upon fsync()'ing
an NFS-mounted, just created, 0-filled 100 Mb file
in less time than it is physically possible.  If
this is a "client" bug, the clients take 95% of
the NFS universe.  I know you're working out the
new specs, but something must be explicitly said
about such blatant violation of intended
semantics.  Without atomicity and locking, already
broken in existing NFS (i.e. "clients" covering
95% of the installations), a dysfunctional fsync()
makes transactional databases' life horrific under
Unix.  They shouldn't be!

  o 	Thanx...
  o 
  o 		ps

-- 
Cheers,
Alexy V. Khrabrov -- www.suffix.com -- Segmentation f%^*&

---

From: Eric Werme USG <[email-masked]>
Date: Fri, 10 Sep 1999 17:26:40 -0400

Linux and Sun both return success upon fsync()'ing
   an NFS-mounted, just created, 0-filled 100 Mb file
   in less time than it is physically possible.

Did you zero file the file byte writing 100 million byte of zeros
or did you lseek to 100 million - 1 and write a byte?

Do you have tcpdump/snoops of the NFS exchange?  Even nfsstat?

  If
   new specs,

cpmp.protocols.nfs is the better forum.

 but something must be explicitly said
   about such blatant violation of intended
   semantics.

Strong language for someone who presents claims and no hard
evidence.  Send me your test program and I'll send you the tcpdump
trace between Tru64 systems.

  Without atomicity and locking, already
   broken in existing NFS (i.e. "clients" covering
   95% of the installations), a dysfunctional fsync()
   makes transactional databases' life horrific under
   Unix.  They shouldn't be!

NFS locking is vastly improved over what it was a few years ago.
Do you have reproducible examples of failures?  Again, c.p.n
is the better forum unless you are making explicit suggestions
for the V4 protocol.

Personally, I think applications with very specific tasks are better served
with custom protocols.  NFS and NFS locking are fine for prototypes, but
once you start unwinding transactions, add logging, atomic lock/unlock of
discontinuous pieces of the database, add server side searches, etc. it's
time for the software equivalent of an ASIC.

	-Ric Werme

--
Eric (Ric) Werme         |  [email-masked]
Compaq Computer Corp.    |  http://people.ne.mediaone.net/werme

---

From: Guy Harris <[email-masked]>
Date: Fri, 10 Sep 1999 15:53:17 -0700 (PDT)

> in less time than it is physically possible.

Are you saying that the "fsync()" call took less time than it would take
to write 100MB of data to the file?

Or are you saying that the time between the point at which you started
writing to the file and the point at which the "fsync()" call returned
was less than the time it would take to write 100MB of data to the file?

In the former case, it could be that the machine started pushing data to
the server before the "fsync()" call (which it's perfectly permitted to
do - there is no guarantee, regardless of whether you're writing to a
local file or to an NFS server, that the system will write *nothing* to
non-volatile storage until an "fsync()" is done).