Subject: Re: [nfsv4] draft-fan-nfsv4-global-namespace-problem-statement-00.txt

From: "Jiaying Zhang" <[email-masked]>
Date: Thu, 7 Apr 2005 12:21:13 -0400

Hi Craig,

I have some questions on your read vs. write considerations.

> (a) In 2.2, you mention that the global namespace will guide clients to 
> the correct replica based on read vs. write considerations.
> I think this is pretty tough or maybe impossible. Posix applications 
> aren't now expressing whether they need R/W access at the
> beginning of transactions, so they might read one file and write another, 
> requiring the most up-to-date version of data for the
By "at the beginning of transactions", do you mean at the time that a client 
mounts a file system?
It is hard to speculate an application's need at the time of mount. But I 
think we may be able to
do this when an application opens a file by looking at open flags. POSIX 
even have three different
open flags to differenciate the consistency semantics an application 
requires: O_SYNC, O_DSYNC
and O_RSYNC. According to POSIX explaination, to get "read-what-you-write" 
consistency, both
O_SYNC and O_RSYNC need to be set.

> reading case. It would be incorrect for the FS to direct the initial 
> "read" part of the transaction to a R/O replica, unless there's
> some magic by which the R/O replicas are up to date with respect to the 
> R/W instance of the data. So I think that other
> mechanisms will be necessary to deal with choosing R/W data vs. allowing a 
> somewhat outdated, but available, R/O replica to
> be used. In this draft, I'd just omit the "access requirement" clause in 
> 2.2.
I agree. We need some specific server-to-server protocol to guarantee 
consistency among replicas.
I am actually working on this. We can use some concurrency control mechanism 
to enforce strict consistency.
But one question is how many applications would require such strong 
consistency (my speculation is for most
applications, ordered writes are enough to guarantee correctness). Do you 
have any examples in mind? Thanks!

Jiaying 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Craig Everhart <[email-masked]>
Date: Fri, 8 Apr 2005 14:36:03 -0400

> O_SYNC and O_RSYNC need to be set.

Really I was saying two things at once.  The basic problem is that Posix
applications don't get to bracket related work with transaction boundaries.
They can open files with flags, as you say, and they can use byte-range
locks
and other techniques to try to indicate their needs to the communication
and
storage systems, but fundamentally they can't say:

      Let's open file A for reading and file B for writing
      Let's read file A, compute some stuff, then write to file B
      OK, as we're done with this, we close files A and B.

In this example, when the filesystem tries to interpret "open file A
for reading," it has no idea of the consistency requirements that the
application really has.  Maybe stale (but not-too-stale) data is good
enough for that open, or maybe it needs up-to-the-minute data.  Sure,
it would be easier to tell if the application were reading just a single
file before updating it, but sometimes the applications aren't written
that way, and there's nothing in Posix to suggest that they should be.

Actually, if you wanted to distinguish application needs at mount time,
that seems promising; there really are read-only mounts and read/write
mounts.  Of course, not that promising, since the interesting and
typical cases aren't doing a lot of read-only mounts, so we really
can't tell.

O_?SYNC flags dictate some semantic requirements on a per-call basis,
but they don't have a lot to do with communication requirements and
serializeability.  I'd think that a byte-range lock could help us out
more here, if they were used.

> have any examples in mind? Thanks!

On this related topic, Peter Honeyman was talking about a replica
management scheme that appeared to provide single-site semantics
rather than allowing data ever to go stale.  Is this what you're
talking about?  What's the difference between "ordered writes" and
single-site semantics?  Are you thinking of providing a scheme in
which writes to all the sites are potentially delayed, so that they
allow other apps to read stale data, as long as the writes are
eventually propagated, in order?

If so, it sounds like a good research topic, but perhaps not a
general update mechanism; there could be applications, and there
probably are applications, that it would break.  This is the kind
of semantic issue that I've seen controlled by an option, like an
option on the functioning of the cooperating servers, controlled by
the administrator, since the applications aren't helping to figure
it out.

What I was imagining for NFSv4 replicas was that they were likely
read-only and generally out-of-date with respect to a source,
though there are exceptions even to that rule, like when the
replicas are really all referring to the same logical data
repository.  Think of having a cluster file system, so that
nodes can all see and operate on what is maintained as the same
file, and having lots of NFSv4 servers all exporting the same
file simultaneously.  The cluster file system is then responsible
for synchronizing each node's view of the file--but that's OK,
because that's what the cluster file system does before breakfast
every day.  What you thus have is a set of coordinated servers
for the same file.  What you're describing is a
less-tightly-coupled version of that, I think, in which you'd
have a lot of explicit data synchronization among the replicas,
each of which would perhaps hold its own copy of the data.

Anyway, back to your question, I guess the answer is that I don't
know how many applications really care about the freshness of their
data.  All I can say is that I bet some do and some don't.
It would be nice if they'd tell the filesystem what they care
about, but they don't.  The usual NFS way has been not to worry
too much about this kind of consistency, I think, but then there
are all the mount options that have let admins worry about it when
they want to worry about it.

> Jiaying

            Craig

Craig Everhart
+1 919 543 2169 (tie 441 2169)