Subject: RE: Issues list for RFC3010

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 23 Sep 2001 10:10:41 -0700

This isn't exactly what you were asking about, but I think
there is a case in which your approach would require some 
identification of who is doing the remove, at least to the
clientid granularity.

Consider a file which is opened by processes on two different
clients, A and B.  Now suppose A does a remove and the 
server does not support a delete on last close mechanism
to it returns NFS4ERR_NO_CAN_DO_USE_THE_OLD_HACK_PLEASE.
So client renames the file.  Now the application on A
does a close so it now removes the file.  But the server
knows that the file is open and would return that same
NFS4ERR_NO_CAN_DO_USE_THE_OLD_HACK_PLEASE and the client
has no way of dealing with that that since he has no way
of knowing when the global last close will occur and, by
hypothesis the server can't do the remove at the appropriate
time either.  And further there is no way for the client to
force the remove immediately either.

If the REMOVE had a clientid parameter, the server could only
check opens for that client and return HACK_PLEASE only if
the file were opened by that client.  Then, in the example
above, the file would actually get removed when it was closed
by A, which would certainly annoy B, but is no worse than what
happens now (i.e in V2 and V3).

An alternative would be to pass a parameter flag that indicates 
this is a hack remove, this_is_the_hack_so_dont_ask_for_the_hack,
which would also force an immediate remove in this situation,
independent of any opens of the file, unless the server dupported
delete on last close.

-----Original Message-----
From: David Robinson [mailto:[email-masked]]
Sent: Friday, September 21, 2001 6:06 PM
To: [email-masked]
Subject: Re: Issues list for RFC3010

> But you didn't address the issue when the server can support it but
> only in cases that it knows that it is what the client wants.  Suppose
> we have a server (say the win2k server) that will disallow REMOVE of
> an OPEN file in the general case.  However, it has the underlying
> capability to allow the client the OPEN, REMOVE, READ sequence.  Your
> proposal doesn't allow the client a method of determining that
> capability.  Maybe Sergey can let us know if this is what his server
> might do?

If I understand this correctly, the question is if there exists a server
that will deny a REMOVE differently depending on if the caller was the
one who OPENed the file or not. For example, you can only REMOVE an OPEN
file if you OPENed it. Sergey, do you know of such a case? I don't.

In any case I would advocate that the server can choose to REMOVE an
OPEN file or not based on whatever criteria it wishes to use. But
if it allows the REMOVE it MUST follow the failure semantics previously
described.

If someone does come up with a weird OS that would require us to
use stateid's or other client state, I will advocate not allowing
REMOVE of OPEN files.

	-David

---

From: David Robinson <[email-masked]>
Date: Mon, 24 Sep 2001 12:10:29 -0500

"Noveck, Dave" wrote:
[Yet another insightful comment]

I am inclined to try and keep the protocol as simple as
possible in these scenerios and accept that one of the
goals of NFSv4 is to NOT provide a distributed consistent
namespace.

The scenerios with a single client operating on an open
file have been discussed and appear to be fairly clear.
The interesting case appears to be when a server will
not remove an open file and a client other than the
one doing the remove has the file open.

If the server has a file that is open by some client,
or even a local open, it will fail the remove regardless
of whether the remove is from a client doing the temp
file trick or some hapless user doing a /bin/rm of the
file. In either case the remove will fail and the server
should respond with an error indicating it failed due to
the file being open and not some generic permissions problem.

In the latter case we can design a complex mechanism to
create a remove on last global close, but I don't think
it is worth the effort. The client may choose to do some
complex tracking, but as with other parts of NFS, sometimes
the server just won't let the client do what it thinks
it should be able to. So the client must issue an
error to the caller and let the chips fall where they
may.

In the former case, the client knows it is trying to
remove an open file and must assume the removed failed
because it had the file open and has no way of
determining if some other client also has it open as
well.  So it does the old rename hack.  When it finally
attempts to do the remove after it closes the file, it
is in exactly the same situation as described above,
a client removing a file it doesn't have open but the
server is failing the request because it thinks it is still
open. Like the previous case the client must just
determine what is acceptable behavior and what to
do with the application (e.g. return an error on close,
silently just continue. or fork a thread to keep trying
to remove the file).

We could add complexity like adding a clientid to
remove, but this adds a bunch of server state
and troublesome recovery semantics. Realistically
only the client knows what it wants and
trying to communicate the multitude of possibilities
to the server through a couple flags doesn't seem
worth the effort. We may have some orphaned files,
but that already happens with today's rename hack.

So in summary, while Dave's scenerio is valid, I don't think
it is any different than other situations where a server
may refuse a remove. From a protocol specification perspective,
if the server cannot remove an open file, it should
return NFS4ERR_NO_CAN_DO_FILE_OPEN and it is up
to the client to determine what it thinks is best.  It
can choose to error to the application, do the
rename hack, keep polling the server with removes,
or reboot, whatever it thinks is best. If the
server does support removal of an open file it
must track the number of opens and provide the
previously described recovery semantics.

	-David

---

From: "Sergey Klyushin" <[email-masked]>
Date: Mon, 24 Sep 2001 14:07:37 -0400

Here is how OpenFile/CreateFile works on Windows (locally):
1. OpenFile/CreateFile has one more share attribute FILE_SHARE_DELETE
1.1 If file is opened WITH FILE_SHARE_DELETE, than request from ANOTHER
application to REMOVE file will succeed.
File won't be removed immediately, but will be removed on the last CLOSE
request.
New OPENs will fail (with exception OPEN to DELETE).
File still will be listed in directory
Attempt to create file with the same name will fail.
1.2 If file is opened WITHOUT FILE_SHARE_DELETE, than request from ANOTHER
application to REMOVE file will FAIL.

2. OpenFile/CreateFile has lots of flags one of which is
FILE_FLAG_DELETE_ON_CLOSE (kind of intention OPEN,REMOVE,READ)
If this flag is specified at OPEN/CREATE, than file be deleted on last
CLOSE.

3. File has attribute DELETE_PENDING, that explains application why it could
not be OPENED other than for deleting.

I am trying to implement NFS as much close to local (Windows) OS as
possible, but some operations could not be
implemented. I understand that NFS protocol can't cover all OSs, but if we
could cover as much local OSs as possible,
we'll benefit it later.

Sergey

    > From: Noveck, Dave [mailto:[email-masked]]
    > Sent: Sunday, September 23, 2001 1:11 PM
    > To: 'David Robinson'; [email-masked]
    > Subject: RE: Issues list for RFC3010
    >
    >

---

From: "Sergey Klyushin" <[email-masked]>
Date: Mon, 24 Sep 2001 15:13:07 -0400

> From: Sergey Klyushin [mailto:[email-masked]]
    > Sent: Monday, September 24, 2001 2:08 PM
    > To: Noveck, Dave; 'David Robinson'; [email-masked]
One more detail. If FILE_FLAG_DELETE_ON_CLOSE was specified on OPEN,
than subsequent open requests for the file will fail, unless
FILE_SHARE_DELETE is used.

    >     > this is a hack remove,
    > this_is_the_hack_so_dont_ask_for_the_hack,

---

From: Spencer Shepler <[email-masked]>
Date: Mon, 24 Sep 2001 20:41:55 -0500

Sergey,
  It seems, from your description, that the windows functionality is
different from Unix in that when a Unix application removes the open
file, it is removed from the namespace.  An open() can then succeed
with the same name as the removed file without restriction with
respect to the impending close() of that removed file.
  Also from your description, the windows client can emulate
everything about remove on close except for failing OPENs from other
clients.  Have you found that this set of options is used often in
windows?
  Since we seem to have this fundamental difference it isn't clear to
me which (if either) piece of functionality we should attempt to
provide.  
  Should we provide the simple functionality that David has suggested?
I am not sure that many server's will implement it.  Should we wait
until a minor version to consider adding a more complex REMOVE op?

Spencer

-- 

- Spencer -

---

From: "Sergey Klyushin" <[email-masked]>
Date: Tue, 25 Sep 2001 10:47:56 -0400

> From: Spencer Shepler [mailto:[email-masked]]
    > Sent: Monday, September 24, 2001 9:42 PM
    > To: Sergey Klyushin
    > Cc: [email-masked]
    > Subject: open/remove sequence
    >
    >
    >
    > Sergey,
I'll ask Dan to comment his Client implementation
    > until a minor version to consider adding a more complex REMOVE op?

David's scenario (simple):
1. Client (C1) OPENs file
2. Client (C1) asks to REMOVE file
3. Server refuses to REMOVE OPENed file
4. Client (C1) renames file to .nfsxxxx (question of non volatile file
handle)
5. ANOTHER client (C2) asks to REMOVE .nfsxxxx (without OPEN)
6. Server refuses, because file is OPENed
7. Client C2 has no idea why REMOVE failed (possible to solve with C1
SETATTR 000 after RENAME,
but C2's request still may come between RENAME and SETATTR)

Another one:
1. Client (C1) OPENs file
2. Client (C1) asks to REMOVE file
3. Server refuses to REMOVE OPENed file
4. Client (C1) renames file to .nfsxxxx (question of non volatile file
handle)
5. ANOTHER client (C2) asks to OPEN .nfsxxxx -> Server responds OK
5. Client (C2) asks to REMOVE .nfsxxxx
6. Server refuses, because file is OPENed
7. C2 RENAMEs .nfsxxxx to .nfs1xxx
8. C1's LOOKUP for .nfsxxxx will fail
8. C1 CLOSEs file using file handle
9. C1 asks to REMOVE .nfsxxxx. Get's back to such file (by name,
but by file handle file still exists -> possible caching problem)

One more:
1. Client (C1) OPENs file
2. ANOTHER Client (C2) RENAMEs file to new_file_name
3. Client (C1) asks to REMOVE file -> failed: file_not_exists. C1 can't do
REMOVE on CLOSE

Almost the same behavior if Server allows REMOVE of OPENed files and makes
RENAME trick locally
Another big question when local OS REMOVEs OPENed file.
From my opinion this approach was fine for NFS v2 and v3, when we did not
have OPEN procedure, but with NFS v4
we may protect ourselves with OPEN operation.
As you said the Client is the one who knows if OPENed file could be REMOVEd,
but Server has no idea about Client's
intention

Windows scenario:
Client wants to use file as temporary, remove it on close and does NOT allow
another clients to remove it
1. Client (C1) OPENs file with DENY_REMOVE, REMOVE_ON_CLOSE (add flags to
NFSv4 protocol)
2. ANOTHER client (C2) asks to REMOVE file -> gets back ERROR, that could be
translated to upper level as ERR_DENIED
3. C2 may OPEN file (question of SHARE mode)
4. C2 can't CREATE file with the same name (file exists and it's true)
5. C1 CLOSEs file
6. Server REMOVEs file

Client wants to use file and does allow another clients to remove it
1. Client (C1) OPENs file WITHOUT DENY_REMOVE
2. ANOTHER client (C2) asks to REMOVE file -> gets back OK or
OK_WILL_BE_DELETED_ON_LAST_CLOSE
3. C2 may OPEN file (question of SHARE mode)
4. C2 can't CREATE file with the same name (file exists and it's true)
5. C1 CLOSEs file
6. Server REMOVEs file

The problems
1. OK_WILL_BE_DELETED_ON_LAST_CLOSE
2. REMOVE_ON_CLOSE state should survive reboot on Server

From my opinion, Windows approach has less problems

Sergey

---

From: David Robinson <[email-masked]>
Date: Tue, 25 Sep 2001 11:16:27 -0500

> but C2's request still may come between RENAME and SETATTR)

I am willing to let this happen, I believe it is rare and the
problem exists today with no major complaints.

> but by file handle file still exists -> possible caching problem)

Same as before, problem exists today with no major complaints.

> 3. Client (C1) asks to REMOVE file -> failed: file_not_exists. C1 can't do

Definately clients today have the third part remove causing unexpected
failoure problems.  Without a namespace consistency protocol we have
no hope in solving this one.

> Another big question when local OS REMOVEs OPENed file.

As I said before, undefined behavior, most likely bad.

> we may protect ourselves with OPEN operation.

I think the OPEN procedure will allow servers to be smarter and better
handle removal of open files, but not perfectly.  I just see it
as a small corner cases that has not be a big problem.

> 1. Client (C1) OPENs file with DENY_REMOVE, REMOVE_ON_CLOSE (add flags to

If these are "standard" windows open flags we should consider adding
them.
There semantics would need to be clearly defined and check to see if
they would cause conflicts with the Posix model of the directory
controling remove permissions.

> 2. REMOVE_ON_CLOSE state should survive reboot on Server

In my previous e-mail, I stated I already believe that a server
will have to maintain state to handle remove on close if
it supports removal of open files, so I don't see this as
a problem.

In reading Sergey's description of windows behavior, it seems
to be a good idea to add DENY_REMOVE and REMOVE_ON_CLOSE if
the semantics make sense (interesting question is if it
prevents actual removal of the file or any of the possibly
many hard links?). The Unix/Posix clients are still faced with
problems, they don't know at open if they will be wanting
a remove on close or have the capability to express DENY_REMOVE.
But those problems exist today and after 2+ years of protocol
design people are only now wondering if it might be a problem,
which to me proves it is not a real industry problem.

Question back to Sergey, if a Windows client requests REMOVE_ON_CLOSE
or DENY_REMOVE and the server fails because it will not allow
it (either unimplemented or not part of the file system semantics)
how will the client handle it? Will it error to the application or
will it be able to set some client state that will issue the
remove after it closes the file?

	-David

---

From: "Sergey Klyushin" <[email-masked]>
Date: Tue, 25 Sep 2001 15:10:12 -0400

> .......
    > .......
    > ..........
    > ...........
If we may remove as much minor problems as possible now, we won't need to
look for customer's complains later

    > the semantics make sense
SHARE_DENY_REMOVE, FLAG_REMOVE_ON_CLOSE and ATTR_PENDING_DELETE are standard
for Windows
and used a lot in Windows Applications, including MS Word, Excel, etc.
It would really help Windows implementation of NFSv4 and
I believe that NFSv4 itself will benefit from these new flags.

    > (interesting question is if it
    > many hard links?).
If file has more than 1 hard link, this semantic will remove only one link

    > The Unix/Posix clients are still faced with
You mean current applications. Yes. In future they could be changed.
But even now Unix clients can benefit from "Windows" approach:
OPEN (acces_mode, DENY_NONE) - I believe will be "standard" UNIX call for
first NFSv4 implementations
REMOVE (file_name) - we don't need RENAME here
READ
CLOSE - file removed by Server.

    > which to me proves it is not a real industry problem.
We raised this question exactly about 2 years ago. The question was put on
hold up to now.
Another HUGE question for Windows implementation of NFSv4 is
OPEN_UPGRADE/OPEN_DOWNGRADE.
We also point this problem about 1 year ago. OPEN_UPGRADE/OPEN_DOWNGRADE
almost impossible to implement
correctly on Windows (the same like SUN found that current stateid model
does not fit Solaris file system,
and we are going to change the model now)

    > remove after it closes the file?
Let's ask Dan to comment it

Sergey

---

From: "Dan Trufasiu" <[email-masked]>
Date: Tue, 25 Sep 2001 17:40:54 -0400

Hi David,

One of the goals of the NFS version 4 was to make the protocol less UNIX
dependent. And all of us agree that a lot was done in the protocol design to
address Windows concerns. There are still some problem that we can not
implement correctly and we will like to investigate the possibility to fix
them. You are correct, we have the protocol for 2+ years, but as you have
seen the "todo list" has 135 issues, some of them being opened for a long
time.

The native Windows procedure to delete a file is the following:
1) Open File with the DesiredAccess "DELETE" flag set,
2) Mark the file for delete (Set Information File  with
FileInformationClass = FileDispositionInformation) this means: Sets
DeleteFile to TRUE  so the file can be deleted when Close File is called to
release the last open handle for the file.
3) Close the file when the delete file operation is supposed to be
performed.

There is no way that we can use the NFS Server for this procedure in NFS V3.
And for this reason we try to emulate it. When I receive the request to mark
for delete I ask for the file attributes and I try to decide using the
attributes if a future REMOVE request will succeed or not. If my decision is
wrong and a file can not be deleted at close the Windows will not process
the CloseFile's error code. The result is a strange behavior when the file
disappears and reappears. We have a lot of customers that complained about
this behavior and our response is that the NFS protocol can not handle every
Windows feature. Some of them understand, but some of them move to other
solutions. What we try to do now is to limit as much as possible the people
that move to other solution because the new protocol can not implement some
very popular Windows features.
You asked me what the Windows client will do if the request to
REMOVE_ON_CLOSE is refused by the server? We will do what we are doing right
now: We will have the same funny behavior.

I'm sure that there are some Windows operation that are very difficult to be
implemented in UNIX. I don't think that the introduction of the DENY_REMOVE
flag can qualify as a impossibility.

Thanks

Dan

> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of David Robinson
> Sent: Tuesday, September 25, 2001 12:16 PM
> Subject: Re: open/remove sequence
>
>