Subject: RE: mmap and locking (was "Caching proposal, take 2 (LONG)")

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 10 Aug 1999 12:23:42 -0700

> I've looked through your proposal, and I'm afraid I'm having problems
> getting a handle on it.  So I'll just ask: what are the implications
> of your proposal for applications that want to mmap a file,
> particularly when the file is locked?

I've always felt that mmap on distributed file systems was an interesting
problem (as in "May you live in interesting times").

Logically, mmap is a form of caching in which the OS gives up the ability
to (without heroic efforts) determine what bytes the actual user is reading 
or writing, except to page granularity.

The short answer is that my proposal, from a functional point of view
leaves things mostly where they are in v3.  Given the difficulties
inherent in mmap, and the restrictions on callbacks that v4 has to
live with I don't think we have much alternative.

It may be that delegation would be a performance help in many situations,
depending on the details of what you chose to implement.  Knowing that
no other clients are doing anything with a file is a powerful basis for 
lots of optimizations having to do with locking and I suppose this
would be true of locking with mmap as well.

Given that v4 has additional features (share reservations and mandatory
locking), the answer above is only approximate.  I'll give you a more
complete answer below.

> The reason I ask is that the Solaris v2/v3 client completely disallows
> concurrent mmapping and locking of an NFS file.  This is to avoid
> problems such as those that you discuss in Section A.4.2, where a
> client needs to avoid writing the entire page if only part of the page
> was locked.  This restriction will likely be relaxed some in Solaris
> 8, but we're getting customer requests to make mmap and locking play
> together in all circumstances.

The problems are similar in being related to page or buffer granu-
larity but I think they are a bit different logically.  In the spec
I deal with the fact that you may not write more than you have locked
to the server (and that this may pose some technical difficulties)
while for mmap the problem is that the client OS may have no way
of knowing what bytes are being written at all.

For v3 clients the typical practice (for normal non-mmap IO) was to do 
the IO synchronously.  This would still work under my v4 caching
proposal since it is a limiting case that (trivially) follows the 
rules that I lay out.

The problem for mmap is that if you are not prepared to do each store
synchronously over the network (and nobody is) then you have no 
occasion to do the actual synchronous writes.

For well-behaved programs (i.e. those that don't read or write 
outside the areas thay have properly locked), the rules in the
caching proposal would allow you to do the writes at unlock time.
This doesn't address the issue of detecting and dealing with 
non-well-behaved programs but I think that is purely a client-OS
issue.

> 
> I don't have a problem with telling these customers that they're not
> going to get what they want.  (In the general case they are asking for
> complete cache consistency.)

Well, they're not going to get complete cache consistency in v4, 
unless there is a massive shift in direction.  

> But I want to understand what they
> *would* get.

The precise "what" depends to some degree on what the client OS
does but I think that you would get:

     The ability to do the support well-behaved mmap users of
     advisory locking by doing writes, revalidation and any
     necessary invaldiations/reads at lock and unlock time.

     In other respects, I believe things are about as they
     were in v4. 

> For example, would the client ever have to revoke a
> previously granted mmap request?

No.  If a delegation is recalled, then either the file
is open or it isn't.  If it isn't open then I assume there
are mmap'ers to worry about when you flush modified data
to the server.  If it is open (for write), then you don't 
to flush the data.  In no case do you have to revoke
an mmap.
  
> And I'd like to make sure the
> Working Group remembers mmap when it considers how much caching v4
> should support.

We're not supporting all that much caching.  I'd like to support
more, but schedule and the callback restriction are tough to get
around.

Here's a little bit more detailed answer on v4 caching and mmap.
You need to distinguish share reservations, advisory record
locks and manadatory record locks.

For share reservations, assuming that mmap's are bracketed by 
OPEN/CLOSE then everything should work fine.  The OPEN may
be denied if someone else opened it with a non-zero DENY
value but there are no mmap-specific issues.

For advisory record locks, I think you are OK as long as the
users are well-behaved.  Enforcing good behavior on 
non-page-bounded locks is difficult but that isn't an
nfs issue (Whew!).  Also, it isn't clear what happens
when users aren't well-behaved and this isn't enforced
but that was undefined in v3 and remains undefined in v4.

For mandatory record locks, there is an issue with mmap.
In the current v4 spec, there is no way for the client
to know that locking was mandatory so that this just 
wouldn't work right.  Within my caching proposal the
client would know to enforce mandatory locking which
can be done fairly easily for page-bounded locks but
is left as an exercise for the client implementor when 
partial pages are involved.  Of course, local file
systems have exactly that same enforcement issue.

---

From: Mike Kupfer <[email-masked]>
Date: Wed, 11 Aug 1999 13:45:15 -0700

Let me see if I understand your response.

1. If the client has a write delegation, the client can write back
whole mapped pages, even if there are locks that cover partial pages.

2. If the file doesn't have any locks, the client must write dirty
mmapped pages back on close, and it can write them back earlier if it
chooses.  Writing whole pages is allowed.  If there is sharing, and a
write from the client clobbers a write from another client, it's the
application's fault for not using locking.

3. If the file is locked and the client does not have a write
delegation, it must write back exactly the contents of the locked
region when the region is unlocked (or the file is closed).  If the
region covers a partial page, the client must not write back the
portion of the page that was not covered by the lock.

If that's not what you meant, let me know what I missed.  It looks
reasonable to me (though I'm not the person who will have to implement
point 3 :-)).

mike