Subject: RE: fsync() fails under NFS, right?

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 14 Sep 1999 11:04:10 -0700

> I was away from my mail, but want to thank all who
> responded.  For us, the issue is very crucial, and
> it is interesting that it doesn't look as such to
> many NFS folks.  The "educated guesses" about our
> model are remarkably correct.

I can see that it is crucial for fsync to work.  I 
believe that there is no sense of crisis precisely 
because nobody has presented clear evidence that 
there is any problem.

> What surpises me is that I get an impression
> that most NFS folks assume fsync() is supposed to
> work under NFS.  However, a dilettante I am, only
> having read Tanenbaum book, I always assumed NFS
> is stateless -- it doesn't maintain cursors for
> each user, then how can it do fsync()??  Flush all
> I/O on the file?

When people say NFS is stateless (sort of true for v2 
and v3, not true for v4), they are referring to the 
protocol between the client and the server.

This statelessness prevents the server from keeping 
per-user or per-open-file state.  It doesn't prevent 
the client from keeping such state in the same way 
that it does for other file systems.

A client may keep track of what data has been written 
to a file by a given user and only flush that on 
fsync.  It may also flush all of the modified blocks
for a file no matter who wrote them since the semantics
of fsync allow you to write more data than you have to.
It is often difficult keeping track of which users have
written which blocks so sending all the modified ones
is a common strategy.  The issues are the same for 
nfs and for local file systems.

> Thus, I'd like to hear what do NFS people say
> think -- fsync() must work under NFS, if it
> doesn't, it's a particular client bug, a correct
> server allows fsync() to work, and there's no
> theoretical problems preventing NFS model from
> having totally operable fsync().  Is that right?
 
No theoretical problems.  The server doesn't see
fsync as an operation.  It sees the WRITE requests
that the server should generate (and possibly a COMMIT).
A server would have to try very hard to make fsync not 
work.  If it is not doing WRITE and COMMIT's properly,
lots of stuff wouldn't work.  It is easier for a 
client to have an fsync bug but if you think you 
have one, you need to demonstrate it with a 
small test program and a packet trace showing that
the correct WRITE requests are not in fact being 
issued.

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 14 Sep 1999 14:34:26 -0400

Alexy Khrabrov wrote:
    
   > many NFS folks.

It does, however, there's a saying that "extraordinary claims
need extraordinary proof".  While we all can readily conceive
of fsync failures (it wouldn't even be extraordinary), you have
presented no proof at all.  We're still waiting.

Dave Noveck replied:

The other reason it may appear that we are disinterested is because
this is a mail list for the discussion of future directions for NFS
protocol development.  No one has brought up any protocol changes necessary
to support fsync() because we all think the V2, V3, and V4 protocols
adequately allow fsync() to be implemented using the protocol.  You're
talking to the wrong audience!

I can't speak for my collegues, but I spend more time than I need
(or should?) answering NFS questions on comp.protocols.nfs and
comp.sys.dec.  That can often save a lot of time instead of going
through support depeartments, especially if two vendors are involved.
On the other hand, we were hired for developing new, working NFS
code.  The support engineers were hired to handle most any problem
a customer encounters and generally have a broader knowledge of
pitfalls.  They can also help you collect traces for study.  We
can't give you that much time.

To the best of my knowledge, Tru64 NFS properly implements fsync(2).
I've read the code, I've studied tcpdump traces, I've timed transfers,
I've counted disk reads and writes.  My rivals and collegues have
done the same.  I've forgotten what vendors you're using.  I admit
I'd be delighted to know if Sun or HP or NetApp had a broken fsync(2).
But I'd be pretty surprised.

	-Ric Werme

---

From: [email-masked]
Date: Tue, 14 Sep 1999 14:53:07 -0400

The consensus seems to be that there're no
theoretical obstacles for fsync(2) to work on a
correct implementation of the NFS protocols as
understood here.  That pretty much answers my
question, for which I am grateful.  The timing I
see on my Linux system tells me something is
wrong.  I'm still trying to see, what...

-- 
Cheers,
Alexy V. Khrabrov -- www.suffix.com -- [email-masked]

---

From: [email-masked]
Date: Tue, 14 Sep 1999 16:31:29 -0400

Dave Noveck states --
  o It sees the WRITE requests
  o that the server should generate (and possibly a COMMIT).
  o A server would have to try very hard to make fsync not 
  o work.  If it is not doing WRITE and COMMIT's properly,
  o lots of stuff wouldn't work.  It is easier for a 
  o client to have an fsync bug but if you think you 
  o have one, you need to demonstrate it with a 
  o small test program and a packet trace showing that
  o the correct WRITE requests are not in fact being 
  o issued.

It occurred to me this is the meat of the
argument.  You say it's all client business.  OK,
my client wants to do fsync().  Now your server
receives WRITE requests.  HOW does my client know,
whether your server WROTE THROUGH to disk or
CACHED my block?  How can I be sure that if your
server crashes, the thing written before an
fsync() doesn't die asleep in that remote cache?

It seems to us our NFS servers have huge caches
and silently swallow our small 100 MB file.  :-)
(All our machines, Linux or Sun, start at 0.5 GB
RAM.)

---

From: Brent <[email-masked]>
Date: Tue, 14 Sep 1999 13:27:03 -0700

Alexy,

This is a good discussion, but you're leaving out all the
folks on the comp.protocols.nfs newsgroup.

This mailing list is specifically for NFS version 4
discussions - not general NFS.  So please, no more
fsync postings unless they're related to NFS v4.

Please continue on comp.protocols.nfs.

	Thanks
		Brent

---

From: [email-masked]
Date: Tue, 14 Sep 1999 16:46:46 -0400

Thanks.  I didn't realize that discussing
transactional issues/improvements does not belong
here.  Sorry.  Bye.

---

From: Guy Harris <[email-masked]>
Date: Tue, 14 Sep 1999 13:36:36 -0700 (PDT)

> Now your server

Unfortunately, there's no NFS request to ask whether a server has bugs,
nor is there any way to automagically detect servers that lie about
their capabilities, so, ultimately, your client has no way of knowing
for *certain*, and there's nothing NFS V4 can, ultimately, do to provide
a 100% certain guarantee.  (For that matter, there's nothing NFS V4 can
to do prevent a meteor from hitting the machine room and crushing the
server, or to do to ensure that the system administrators have been
doing backups....)

However, if the server is following the spec for NFS V2 or NFS V3, the
V2 spec says:

   All of the procedures in the NFS protocol are assumed to be
   synchronous.  When a procedure returns to the client, the client can
   assume that the operation has completed and any data associated with
   the request is now on stable storage.  For example, a client WRITE
   request may cause the server to update data blocks, filesystem
   information blocks (such as indirect blocks), and file attribute
   information (size and modify times).  When the WRITE returns to the
   client, it can assume that the write is safe, even in case of a
   server crash, and it can discard the data written.  This is a very
   important part of the statelessness of the server.  If the server
   waited to flush data from remote requests, the client would have to
   save those requests so that it could resend them in case of a server
   crash.

so if the client is talking to an NFS V2 server *that follows the spec*
(not all do, and there's no way to inquire whether it's doing so), you
don't get a reply until the data is in non-volatile storage, and the V3
spec says:

	4.7 Synchronous modifying operations

	   Data-modifying operations in the NFS version 3 protocol are
	   synchronous. When a procedure returns to the client, the
	   client can assume that the operation has completed and any
	   data associated with the request is now on stable storage.

	4.8 Stable storage

	   NFS version 3 protocol servers must be able to recover
	   without data loss from multiple power failures (including
	   cascading power failures, that is, several power failures in
	   quick succession), operating system failures, and hardware
	   failure of components other than the storage medium itself
	   (for example, disk, nonvolatile RAM).

	   Some examples of stable storage that are allowable for an NFS
	   server include:

	   1. Media commit of data, that is, the modified data has
	      been successfully written to the disk media, for example,
	      the disk platter.

	   2. An immediate reply disk drive with battery-backed
	      on-drive intermediate storage or uninterruptible power
	      system (UPS).

	   3. Server commit of data with battery-backed intermediate
	      storage and recovery software.

	   4. Cache commit with uninterruptible power system (UPS) and
	      recovery software.

	   Conversely, the following are not examples of stable
	   storage:

	   1. An immediate reply disk drive without battery-backed
	      on-drive intermediate storage or uninterruptible power
	      system (UPS).

	   2. Cache commit without both uninterruptible power system
	      (UPS) and recovery software.

	   The only exception to this (introduced in this protocol
	   revision) is as described under the WRITE procedure on the
	   handling of the stable bit, and the use of the COMMIT
	   procedure.  It is the use of the synchronous COMMIT procedure
	   that provides the necessary semantic support in the NFS
	   version 3 protocol.

(where 4.7 either isn't entirely true, or doesn't consider an
asynchronous WRITE to be a "data-modifying operation", but does consider
a synchronous WRITE or a COMMIT to be a "data-modifying operation)", so
if the client is talking to an NFS V3 server *that follows the spec*
(given that the server *can*, if the client indicates in the WRITE
request that the write needn't be synchronous, reply before the data
makes it to non-volatile storage, I suspect few, if any, NFS servers do
"dirty evil cheating asynchronous writes" for V3, unlike V2 where some
can be configured to do so, and some even do so *by default*), you don't
get a reply until the data is in non-volatile storage.

As indicated, there's nothing we can do in the protocol to improve this
- V3 includes all the mechanisms to do "safe" async writes, so that a
server is less tempted to do unsafe async writes, but the protocol
itself cannot enforce Good Behavior.

> and silently swallow our small 100 MB file.  :-)

Perhaps, perhaps not.  I'd have to see some indication that, say, no
disk (or NVRAM) writes had been done on the server until the "fsync()"
to be convinced that the problem is that the server is cheating and not
ensuring that the data made it to stable storage before replying to a
write.

---

From: [email-masked]
Date: Tue, 14 Sep 1999 17:01:31 -0400

I'm back.  ;-) Dave, Guy -- thanks for your clear
and informative responses.  Unlike TIAs, these are
real answers!

---

From: Eric Werme USG <[email-masked]>
Date: Tue, 14 Sep 1999 16:54:32 -0400

CACHED my block?

If the server is properly written and the client sends V2 writes or V3
synchronous writes, the server WILL NOT REPLY UNTIL THE DATA IS ON THE DISK. 
Period.  If you are using a stock Linux server, you are using NFS V2 and I
believe the Linux server does not wait for writes to reach the disk before
replying.  That's why many people see much better performance on Linux
servers than from vendors you pay money to.  Fast, cheap, or reliable - pick
two.  If you're staking your business on Linux, you need a QA department
patterned after the major vendors' departments.  This mail list is
definitely not a Linux support hotline!

  How can I be sure that if your
   server crashes, the thing written before an
   fsync() doesn't die asleep in that remote cache?

In the above examples, the client will retransmit the write until our server
reboots and finally gets the data to disk and then replies.  Um, our servers
don't crash.  :-)

If you're using V3 async writes, then the client has no way of telling
if the data is in the cache or on the disk.  However, fsync() will block
until all the data is on the disk.  The client will send a commit at
some point (e.g. fsync()).  A "write verifier" is used to determine
if the server rebooted between replying to the write and to the commit.
If the server rebooted, then the client will retransmit the writes that
commit may have missed.  Either synchronous writes will be done, or
asynchronous followed by another commit will be done, the latter checking
the write verifier again.

We've been through this many times now, please learn this, I'm not typing it
again, but I might mail you the V2 and V3 spec....

IT'S TIME FOR SOME PROOF.  Take a Sun-Sun system, watch writing a
small file with snoop.  It will work, you will see substantial delays
between commit request and response.  Linux client, Sun server will
probably work if fsync() isn't a no-op.  Linux server will probably
reply too quickly in all cases.

If you want to prove a failure, start writing a big file, and turn off
the server.  Turn it back on and reboot.  Let the client finish, then
verify the file on the server.  Try again, but push reset during the
write.  Repeat until you have a corrupted file, send it and
the snoop/tcpdump trace to your vendor's support department.

	-Ric Werme

---

From: [email-masked] (Brian Pawlowski)
Date: Tue, 14 Sep 1999 13:28:05 -0700 (PDT)

Please take this discussion to comp.protocols.nfs now.

Brian Pawlowski

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Wed, 15 Sep 1999 07:17:21 -0400

From: Eric Werme USG <[email-masked]>
   Date: Tue, 14 Sep 1999 16:54:32 -0400

For what it's worth, modern Linux systems use a kernel-level NFS server
which correctly implements the syncing semantics demanded by the NFS V2
specification.  (NFS V3 support is currently underway but its not yet
completely supported.)  

I'm not sure [email-masked] was thinking, but rest assured that people
who are actually working on the Linux NFS server code are well aware of
these issues, and in fact by default correctly support the NFS
synchronous write semantics.

The user-level NFS server which Linux used to use was slow, inefficient,
and deserved derision vis-a-vis its synchronization semantics.  It was
certainly not enterprise ready in any sense of the world.  However, it's
not fair to malign Linux's NFS implementation based on outdated
information.

The comment about nfs4-wg not being a Linux support hotline is
definitely well taken, though.  This is list is also certainly not a
place for educating people who don't understand how NFS works.  Trust
me, the folks who are really working on Linux NFS don't suffer from this
problem.  I've talked to them, and they are not without clue.

					- Ted
					VA Linux Systems

---

From: [email-masked]
Date: Wed, 15 Sep 1999 13:07:25 -0400

The tone of the message by "Theodore Y. Ts'o"
implies a strange, unjustified superiority of some
Linux developers, which probably points to real
inferiority complexes of the authors or
carelessness of speech, either of which wouldn't
encourage my response.  However, since some
inappropriate things have been said to the group,
and an important technical culprit has been let
out of the box by Gen. Ts'o, I'd like to analyze
further.  Please pardon my follow-up, but that
message can not be put away for the culprit,
philosophical implication described below.

Our NEC group relies on Linux for development of
serious distributed systems, and one point of the
respected self-proclaimed Linuxoid's posting is
very important to notice.  It is aggressive kernel
optimization.

One of the things we noticed and investigated were
TCP protocol problems under Linux.  It seemed that
the kernel does buffer stuff on behalf of
processes in a way not fully faithful to the TCP
specs.  A process would get an OK status after
sending messages while they are actually in the
kernel buffer and the destination process could be
dead.  The only way around it was to get the
recipient send an ack back.  Clean TCP spec would
allow teh sender to receive the status and that
would be enough.  We discussed the problem with
Alan Cox at the time.  It may or may not be
corrected by now.  But the fact how subtle it was
and that it was indeed introduced into Linux is
telling.

Notice a similarity in spirit with transactional
semantics -- the ack must come only after
something happens for sure, but Linux didn't
carefully ensure it did indeed happen.

Since Linux aggressively positions itself as the
server platform, Linux folks smell blood in the
water and move ahead with kernel optimizations of
all sorts.  With some waters, some babies also get
thrown out.  

The point of my message is that it is important to
get reality-level code, but not at the price of
cutting corners in such ibscure ways that it takes
seven and a half gurus to understand what's going
on.  If Linux wants to run ahead of the old dogs,
which is happenning, it should pay more attention
to protocol quality, scientific discussion, and be
subject to appropriate criticism.  OK?

---

From: "Theodore Y. Ts'o" <[email-masked]>
Date: Wed, 15 Sep 1999 17:13:00 -0400

From: [email-masked]
   Date: Wed, 15 Sep 1999 13:07:25 -0400

Alexy, 

I've got news for you.  All Unix systems do this; a successful write to
a TCP socket does not indicate that the destination has received the
data.  The kernel will queue the data for retransmission if necessary,
but the write() itself does not block until the other side has ack'ed
the data.  And there is no TCP requirement in its specifications for the
semantics which you describe.

May I suggest you take your misinformation and your Linux bashing to
another forum?  This has nothing to do with NFS V4.

							- Ted

---

From: [email-masked]
Date: Wed, 15 Sep 1999 19:01:01 -0400

I didn't describe the fault in detail, but clearly
indicated that TCP returns were not corresponding
to eventual safe write at all points through.  We
thoroughly checked against the TCP specs and sure
there was a problem.  Your news are no news, and
there's no misinformation.

I am an avid Linux supporter and point out ways to
improve NFS implementation, etc., to make it
better.  There's no Linux bashing; I have it
running on all my machines, and my home and office
networks use Linux and Linux NFS.

Your original material described a technical
agenda of kernel optimization, which we believe
should be done more carefully when implementing
complex semantics such as NFS.  Moreover, such
protocols must be designed with the appreciation
and apprehension of such optimizations.  Thus it
warranted a response based on our experience with
problems of such optimizations, and as being fully
on-topic.

As per your suggestion about other forums, I
received specific pointers from other members,
better phrased and more helpful.  (Of course, I'll
keep rebutting any ad hominem stuff ad infinitum.)
Thanks.

---

From: Eric Werme USG <[email-masked]>
Date: Fri, 17 Sep 1999 19:21:37 -0400

Period.  If you are using a stock Linux server, you are using NFS V2 and 
  I
      believe the Linux server does not wait for writes to reach the disk befor
  e
      replying.

Sigh. I try to gracefully end one discussion that doesn't belong here and
another pops up.  I will post no more on fsync and related Linux until
Alexy comes up with the hard data we've asked of him.

Ted, the post you quote was deliberately simplistic and deliberately
avoided distinguishing between the user level server and the kernel
level server.  I've been a member of the Linux nfs-devel list since
February, and have corresponded with Trond, Olaf, and the list on V3
implementation notes.  So have other people on this mail list.  I have
no idea what version of Linux Alexy is using, he hasn't said.  Given
that knfsd server didn't pass the Connectathon test suite until a week
or so before Connectathon, I figured there is a very good chance that
Alexy isn't running a "Modern" Linux kernel.

Rest assured that several people on this list have made the developers
well aware of the importance of invariant file handles, rename
behaviour, etc.  I wasn't aware that a V2 server did sync all data,
given the number of c.p.n queries about Linux client -> Linux server
being several times faster than going to other vendor's servers.

Apologies for not running the latest and greatest, it's one of the
reason I kept after Alexy for hard data instead of rolling my own.
However, I installed RH 5.2 here soon after Connectathon and so knew
there was little reason to use either the user or kernel level NFS
server.  (IIRC, it wasn't until a week or so before Connectathon that
knfsd passed the Connectathon test suite.)  Once there's a release
with V3 built in, I'll probably upgrade, as a working V2 server isn't
very interesting anyway!

	-Ric

And, apologies to the rest of the mail list for spending time on this
thread when I should be looking for time to be able to post
worthwhile, on-topic posts here.