Subject: Issues list for RFC3010

From: Spencer Shepler <[email-masked]>
Date: Wed, 19 Sep 2001 19:47:46 -0500

I have an updated issues list at:

http://www.nfsv4.org/rfc3010updates/issues.html

Updates will follow.

-- 

- Spencer -

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 09:25:31 -0400

For easier processing of COMPOUND requests, I would like to propose minor
protocol changes.
The idea is to set "int" type parameters before "opaque" in arguments and
results.
There are not much such structures, and I believe it's not too late to
change .x file.
I found following structures (based on RFC 3010 draft-ietf-nfsv4-07)

struct CREATE4args {
	createtype4 objtype;
	component4 objname;
};

struct LOCK4denied {
	offset4 offset;
	length4 length;
	nfs_lockowner4 owner;
};

struct LOCKT4args {
	nfs_lock_type4 locktype;
	offset4 offset;
	length4 length;
	nfs_lockowner4 owner;
};

struct OPEN4args {
	seqid4 seqid;
	uint32_t share_access;
	uint32_t share_deny;
	nfs_lockowner4 owner;
	openflag4 openhow;
	open_claim4 claim;
};

struct open_claim_delegate_cur4 {
	stateid4 delegate_stateid;
	pathname4 file;
};

Sergey

    > -----Original Message-----
    > From: Spencer Shepler [mailto:[email-masked]]
    > Sent: Wednesday, September 19, 2001 8:48 PM
    > To: [email-masked]
    > Subject: Issues list for RFC3010
    >
    >
    >

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 11:31:01 -0400

>Issue03:
>OPEN(CREATE); add NFS4ERR_ISDIR for the case the name to create is a
directory (not a regular file).

>Issue04:
>OPEN(NOCREATE); add NFS4ERR_NOENT for the no entry case.

Should NFS4ERR_ISDIR be used in case OPEN(NOCREATE) and the name to create
is a directory?

Sergey

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 11:41:12 -0400

>Issue08:
>Change of "tag" in COMPOUND4args. At a minimum the server should return the
same tag length/contents that were >included in the request. Alternate
changes are: fixed length tag or no tag at all.

I vote to remove "tag" at all. If it used for debugging - use snoop, if it
used for caching - use something else (xids from RPC)

Sergey

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 12:42:24 -0400

directory (not a regular file).
Should we add the same error to CREATE? I mean if target for CREATE(not dir)
exists and target is directory. Should we add NFS4ERR_ISDIR to all
operations that reference to file (WRITE, etc.)?

>Issue11:
>SETCLIENTID_CONFIRM should use clientid and verifier instead of just the
verifier
>Issue15:
>OPEN_CONFIRM should take the stateid along with the seqid and confirm
verifier
Do we really need both clientid (stateid+seqid) and verifier? Why just
clientid (stateid+seqid) is not enough?

>Issue19:
>Attributes for OPEN(CREATE). The suggestion has been made to add an
attributes parameter for OPEN(CREATE) of a regular file so that the client
would not have to use a subsequent SETATTR to obtain the desired attribute
settings.
Do you mean OPEN or CREATE? OPEN already has attributes, but CREATE doesn't.
From my point of view,
CREATE should also take attributes to set (at least for directory and
symlink) and return bitmap of what attributes were set

>Issue14:
>Introduce method that allows client to determine if server is supporting
POSIX-style file locking or "strict" file locking. Two apparent choices: add
new attribute or return the information as a bit in the results flags field
(rflags) in OPEN4resok.
I think NFSv4 protocol should say in more details what means "POSIX-style
file locking or "strict" file locking"

>Issue16:
>Should a value be added to the fh_expire_type attribute to allow the server
to signify that for replicated filesystems the filehandles are the same
between servers.
It will be very useful for NT. NT won't include this flag to signify that
file handles are different, but other OS could set this flag and make life
of NFS Client much easy

>Issue25:
>Should the "filehandle" attribute be mandatory?
Yes. Almost all operations use current filehandle. What could be done
without filehandle?

>Issue32:
>Section 8.9 states that the server MUST return an error if CLOSE would
result in having locks exist after the CLOSE operation. What error is
supposed to be used?
Also what error should be returned if server can't close file with
outstanding locks?

>Issue33:
>LINK operation needs the NFS4ERR_NOENT added as possible error return.
What error should be returned on "ln f1 f2" request, if f1 and f2 are
already hard linked?

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 14:27:26 -0500

I don't see a proble with this.  I will add it to the issues list.

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 14:27:57 -0500

> Should NFS4ERR_ISDIR be used in case OPEN(NOCREATE) and the name to create

Yes, of course.  Just missed this in the issue; will update.

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 14:29:34 -0500

> used for caching - use something else (xids from RPC)

Well, I have found that in using snoop it is really nice to have a tag
string to give you some indication of what the client is attempting to
accomplish with the string of COMPOUND operations.

My personal preference would be to have a fixed length string.  We
could even make it a regular xdr string type to avoid the utf8
"overhead".

-- 

- Spencer -

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 17:05:05 -0400

>Issue125:
>Client's REMOVE of an open file still requires the use of a RENAME to a
.nfsXXXX file
>followed by a subsequent REMOVE when the client's applications release
final reference to the open file.

I think it's still "shaky". Do you mean the same Client sends REMOVE request
or another?
What if another Client sends READDIR and RENAMEs or OPENs .nfsXXXX?

I have proposal to introduce new file/directory attribute:
FATTR4_MARKED_TO_DELETE.
I believe lots of OS support this attribute locally.

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 17:13:20 -0400

>Issue116:
>In section 8.4 of the spec, it states:
>The following events cause implicit renewal of all of the leases for
>a given client (i.e. all those sharing a given clientid).  Each of
>these is a positive indication that the client is still active and
>associated state held at the server, for the client, is still valid.
>
>   o  An OPEN with a valid clientid.
>
>   o  Any operation made with a valid stateid (CLOSE, DELEGRETURN,
>   	LOCK, LOCKU, OPEN, OPEN_CONFIRM, READ, RENEW, SETATTR,
>   	 WRITE).  This does not include the special stateids of all
>   	 bits 0 or all bits 1.
>
>
>OPEN_DOWNGRADE should be added to this list as it takes a stateid as an
argument.
>SETCLIENTID_CONFIRM should also be added to the list for completeness.

I don't agree with SETCLIENTID_CONFIRM.
1. Until SETCLIENTID_CONFIRM is received there are either no established
states for the CLIENT, or all states will be released on successful
SETCLIENTID_CONFIRM.
2. If client was already confirmed, SETCLIENTID_CONFIRM will be
retransmission or error from Server's point of view.

Sergey

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 16:16:46 -0500

> What if another Client sends READDIR and RENAMEs or OPENs .nfsXXXX?

The issue was that there was no mention in RFC3010 of the standard NFS
implementation practice of renaming a file to .nfsXXXX when an
application had the file open at the client.  We are referring to the
same client.  If another client removes the file while another has it
open, the application at the client where the file is open will
receive an error.  This assumes that the server supports the remove of
a file that is open (Solaris will support this).

> I believe lots of OS support this attribute locally.

So how does this work at the server?  Is this effective across server
restart?  (this isn't required since .nfsXXXX would still exist as
well but it would be good to know).  So the idea is that if the client
has the file open and this attribute is returned by the server, the
client would go ahead with the REMOVE and when the last NFS OPEN is
CLOSEd the file would be removed by the server automatically?

What OSes support this feature?

Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 16:18:47 -0500

> retransmission or error from Server's point of view.

So the larger issue is when does the server start the countdown for
the lease period.  Does the counter start when the client instantiates
its first state at the server via OPEN?  Is it from the
SETCLIENTID_CONFIRM?

I agree that SETCLIENTID_CONFIRM probably isn't appropriate but we
need to decide the meta issue to resolve this correctly.

Spencer

---

From: Brent <[email-masked]>
Date: Thu, 20 Sep 2001 15:29:48 -0700

Spencer Shepler wrote:
> >   :
>   :

I like the idea of a fixed length for the tag.

It seems that its variable length is more of a 
problem than its presence.

	Brent

---

From: "Sergey Klyushin" <[email-masked]>
Date: Thu, 20 Sep 2001 20:16:17 -0400

> Sent: Thursday, September 20, 2001 5:19 PM
    > Subject: Re: Issues list for RFC3010
    >
    >
As far as I remember, RFC says that counter starts on successful SETCLIENTID
request. More than that,
SETCLIENTID_CONFIRM should be send by client within lease period.

Sergey

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 21:48:39 -0500

> operations that reference to file (WRITE, etc.)?

I have added two more issues to include NFS4ERR_ISDIR for WRITE and
CREATE.  There are already other operations listed as needing ISDIR.
Let me know if any others are missing.

> clientid (stateid+seqid) is not enough?

I believe you are correct.  If the stateid is added to OPEN_CONFIRM it
acts like the verifier because it will change over time.  Therefore we
should be able to replace the verifier with stateid. I will update the
issue with the suggestion.

> symlink) and return bitmap of what attributes were set

My mistake.  It should be CREATE.  Will update the issue.

> file locking or "strict" file locking"

Okay.  I will add an issue that we need to add clarifying text.  Maybe
we should just say Unix fcntl() locking vs. Win32 locking. :-)

> file handles are different, but other OS could set this flag and make life

Other opinions?

> Yes. Almost all operations use current filehandle. What could be done

Other opinions?

> Also what error should be returned if server can't close file with

Same question, right?

From 8.9
					"The server MAY free all
   outstanding locks on CLOSE but some servers may not support the CLOSE
   of a file that still has record locks held.  The server MUST return
   failure if any locks would exist after the CLOSE."

Which I read as the CLOSE was not executed (file still open?) and an
error is returned.  We need to define the error and maybe a path for
how the client is going to deal with the state left over from the
failed CLOSE.

> What error should be returned on "ln f1 f2" request, if f1 and f2 are

NFS4ERR_EXIST.  Will add a description to the IMPLEMENTATION section.

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 20 Sep 2001 22:08:14 -0500

> SETCLIENTID_CONFIRM should be send by client within lease period.

I don't think there are comments in the RFC about the lease starting
at SETCLIENTID receipt.  If you find them, let me know. :-)

In any case, if SETCLIENTID starts the lease period, then why wouldn't
SETCLIENTID_CONFIRM count towards renewal of that lease.

We need to make it clear in the discussion of lease periods that if
the client is successful with a SETCLIENTID/SETCLIENTID_CONFIRM
sequence but does not send any other request within the lease period,
the client may (will more than likely) have to do the
SETCLIENTID/SETCLIENTID_CONFIRM sequence again before proceeding.

This may be common.  For example, if a client is rebooted in the
middle of the night and mounts its filesystems from the server and in
doing so does the SETCLIENTID/SETCLIENTID_CONFIRM but has no further
activity it will probably not sit there and RENEW the lease at the
server.  To limit memory usage, the server will free the client's
references and the client will need to reestablish them.

So I will add an item that the lease period start needs to be
clarified and a brief discussion of client recovery after no activity
(if it is not already covered).

-- 

- Spencer -

---

From: Peter Åstrand <[email-masked]>
Date: Fri, 21 Sep 2001 09:46:00 +0200 (CEST)

> > > used for caching - use something else (xids from RPC)
...
> problem than its presence.

Why is the variable length a problem?

My opinion is that the tags should be included in the protocol, in one
form or another. They are really useful, IMHO.

-- 
Peter Åstrand                Telephone: +46-13-29 08 61
Cendio Systems               E-mail: [email-masked]
Teknikringen 3
583 30 Linköping

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 02:58:05 -0500

> form or another. They are really useful, IMHO.

Variable length adds overhead to the decoding of the request.

-- 

- Spencer -

---

From: "Sergey Klyushin" <[email-masked]>
Date: Fri, 21 Sep 2001 10:59:59 -0400

> Sent: Thursday, September 20, 2001 11:08 PM
You are right, I was confused reading
=========================================================================
8.1.2.  Server Release of Clientid
   If the server determines that the client holds no associated state
   for its clientid, the server may choose to release the clientid.  The
   server may make this choice for an inactive client so that resources
   are not consumed by those intermittently active clients.  If the
   client contacts the server after this release, the server must ensure
   the client receives the appropriate error so that it will use the
   SETCLIENTID/SETCLIENTID_CONFIRM sequence to establish a new identity.
   It should be clear that the server must be very hesitant to release a
   clientid since the resulting work on the client to recover from such
   an event will be the same burden as if the server had failed and
   restarted.  Typically a server would not release a clientid unless
   there had been no activity from that client for many minutes.
=========================================================================
I translated "inactive client", "no activity from that client for many
minutes" to
no activity during lease period. Now I realized, that NFS Server should have
4 timeouts
1. Between SETCLIENTID and SETCLIENTID_CONFIRM.
2. lease time for open_stateid
3. lease time for locked_stateid
4. Successful SETCLIENTID/SETCLIENTID_CONFIRM, but no activity from client,
meaning
no operations that allow Server find clientid (OPEN, LOCK, ...). The timeout
could be the same as for 1.

Am I wrong again?

Sergey

---

From: "Sergey Klyushin" <[email-masked]>
Date: Fri, 21 Sep 2001 12:32:00 -0400

> Sent: Thursday, September 20, 2001 5:17 PM
OK for NFS v.2 and v.3 that did not have OPEN, so returning error was OK.
NFSv4: Client opens file with SHARE=deny to prevent everybody else to modify
file's context,
and some other client not only modifying it, but removes it. Is it fair?
Am I correct, that this technique was used for temporary files?

    > What OSes support this feature?
I will provide more details.
1. The proper attribute name should be FATTR4_DELETE_PENDING
2. Meaning of FATTR4_DELETE_PENDING: remove file on the last CLOSE
3. This attribute could be set/unset on OPEN operation or on OPENed file by
SETATTR, providing correct stateid.
SETATTR with stateid of all 0's or of all 1's can't set this attribute
4. REMOVE of OPENed file won't remove file immediately, but will set this
attribute
and the file will be deleted on the last CLOSE
5. OPEN operation on the file with FATTR4_DELETE_PENDING set will fail with
ACCESS DENIED error

Answers to your questions
1. On server restart file will be deleted, because usually Server closes all
OPENED files before restart.
It's possible to provide functionality not to remove file on restart and
restoring FATTR4_DELETE_PENDING.
2. Windows NT supports this feature and it's used by lots of Windows
applications for temporary files
I am not sure about other OS, but there are the gurus for Solaris, NetApp,
Linux, DEC, etc. in our mailing list.

Your comments

Sergey

---

From: David Robinson <[email-masked]>
Date: Fri, 21 Sep 2001 12:04:28 -0500

> 5. OPEN operation on the file with FATTR4_DELETE_PENDING set will fail with

The key question is what is the semantics from the server's
perspective from the compound OP <OPEN, REMOVE, READ>?

One alternative is to simply require the REMOVE of an OPEN
file to fail.  Clients can then use the traditional RENAME
trick and REMOVE after CLOSE.

The second alternative is to allow this sequence.  The interesting
question becomes if the REMOVE actually prohibits and further
access of the file through the namespace using the old name?
I think the answer is yes, the server knows the file is open
so it can rename it to a .nfsXXX or move it to a /nfs_deleted_files
directory.  The result is that the file can only be accessed
via its file handle, all subsequent LOOKUP or OPEN will fail.
And it is possible to CREATE a new file with the old name.  The
server then MUST handle failure in a sane manner, it MUST NOT
delete the file until after the lease interval has expired, but
MAY allow it to last longer if it wants to accept late lease renewals.

One implication of the second alternative is that file handles MUST
NOT be volitile between the REMOVE and the CLOSE including the
possibility of server reboot.  Those servers that cannot meet
this requirement MUST fail the REMOVE and the client may choose
to resort to the old .nfsXXX RENAME trick.

In either alternative, I don't see the value of the
FATTR4_DELETE_PENDING attribute.  It doesn't provide much
useful information to the clients and the servers can
internally track open but removed files.

My opinion is the second alternative above provides the most
flexibility and is not overly complex for servers that wish
to support it.  But still allows simple servers to
implement the first alternative by failing the REMOVE.
(Might need to add a new error code to reflect the response of
"E_thanks_I_would_normally_allow_the_remove_but_sorry_it_is_open".

	-David

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 15:38:52 -0500

> You are right, I was confused reading

I don't think so but I think it might be helpful to state these
differently.

There is the ultimate timeout at the server -- the lease period.
The questions here are: 
	- when does the lease period start
	- when does the lease period RENEWed (what operations)

I would argue that the lease period starts when the first state
operation (OPEN) is executed by the server for that client.  The lease
is then renewed on the defined set of renewing operations.  By
definition, if the lease period expires, the server is free to release
all state and data structures related to that client.

There are comments in the specification about the server's need to
free data structures related to state that is not currently active.
Things like a lockowner reference when that lockowner does not have
any open files.  As has been pointed out, the clientid can also be
freed at the server.  At a minimum, the server should wait for a full
lease period before releasing the clientid (confirmed or not).  As for
the lockowner references, I believe that we have discussed for the
purposes of handling retransmitted CLOSEs that the server should wait
a lease period before releasing that mechanism and therefore the
server should probably wait a lease period before freeing the
lockowner references.  I am not sure if this is too burdensome.

For the case where the duration between SETCLIENTID_CONFIRM and OPEN
is longer than a lease period, the server is allowed to free the
clientid and return NFS4ERR_STALE_CLIENTID as a result of that OPEN.

Does this need to be added to the document in a concise way?  Is this
interpretation meet everyone's understanding?

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 15:42:48 -0500

> "E_thanks_I_would_normally_allow_the_remove_but_sorry_it_is_open".

I like the approach, David.  If we go with something like this, it
seems like we need to allow for the removal of the file by other means
at the server.  For example, by local access to the fliesystem in
question.  I don't think we should place the burden on the server to
implement server-wide protection against removal.

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 15:45:08 -0500

> "E_thanks_I_would_normally_allow_the_remove_but_sorry_it_is_open".

Another last minute thought.  What if the server's normal mode of
operation is that it will not allow the removal of an OPEN file and
therefore needs some special identification that a REMOVE is part of
an applications desire to OPEN, REMOVE, READ.  Do we want to burden
REMOVE with a stateid?  Right answer?  Too heavy weight?

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 15:47:49 -0500

> Am I correct, that this technique was used for temporary files?

Yes.  An NFSv4 server may allow an OPEN(SHARE_DENY_WRITE) from one
client and then a REMOVE of that same file from the same or another
client.

> Linux, DEC, etc. in our mailing list.

Solaris, AIX do not have such a mechanism for holding an open file in
such a way as to allow access to that file across server reboot.  I
believe this is a requirement of this type of functionality as David
has pointed out.

-- 

- Spencer -

---

From: David Robinson <[email-masked]>
Date: Fri, 21 Sep 2001 16:04:32 -0500

> implement server-wide protection against removal.

If I understand you correctly, you are arguing
a file opened via NFS but removed through a different mechanism
(local access or different protocol) has undefined 
semantics? If so I agree completely.  I would encourage
NFS server vendors to implement a best possible solution
that causes the least surprise, but it would not be wise to
mandate a solution.

	-David

---

From: David Robinson <[email-masked]>
Date: Fri, 21 Sep 2001 16:15:29 -0500

> REMOVE with a stateid?  Right answer?  Too heavy weight?

Aghh, no... we have too much exposed state as it is...

The problem only occurs when the client and server have different
semantics for the removal of an open file.  If the
client APIs prohibit it the server will not see an OPEN-REMOVE
unless the client is buggy.  If the client allows it but the server
prohibits it, the server sends back an error and the client
if it wishes to protect the application must try a different
trick like the .nfsXXX RENAME. And lastly if both allow it the
server must maintain internal state that survives a reboot
for error recovery.

What I was trying to do here was come up with a semantic that
doesn't require the server to expose any state to the client.
Sergey's solution looked workable but it forced the client
to track more state. Well technically if the client needs
to perform the RENAME .nfsXXX trick that is state that it
needs to track, but that was solved 15 years ago and already
exists in all OSes that care. The client maintaining the
state that a file is OPEN is sufficient.

Now if a client is accessing a file without using OPEN and
then REMOVEs it, it gets undefined behavior. "Don't do that..."

	-David

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 16:43:31 -0500

> Aghh, no... we have too much exposed state as it is...

:-)

> state that a file is OPEN is sufficient.

But you didn't address the issue when the server can support it but
only in cases that it knows that it is what the client wants.  Suppose
we have a server (say the win2k server) that will disallow REMOVE of
an OPEN file in the general case.  However, it has the underlying
capability to allow the client the OPEN, REMOVE, READ sequence.  Your
proposal doesn't allow the client a method of determining that
capability.  Maybe Sergey can let us know if this is what his server
might do?

-- 

- Spencer -

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 21 Sep 2001 16:45:26 -0500

> that causes the least surprise, but it would not be wise to

Correct.  This is what I was saying.

-- 

- Spencer -

---

From: David Robinson <[email-masked]>
Date: Fri, 21 Sep 2001 17:05:33 -0500

> capability.  Maybe Sergey can let us know if this is what his server

If I understand this correctly, the question is if there exists a server
that will deny a REMOVE differently depending on if the caller was the
one who OPENed the file or not. For example, you can only REMOVE an OPEN
file if you OPENed it. Sergey, do you know of such a case? I don't.

In any case I would advocate that the server can choose to REMOVE an
OPEN file or not based on whatever criteria it wishes to use. But
if it allows the REMOVE it MUST follow the failure semantics previously
described.

If someone does come up with a weird OS that would require us to
use stateid's or other client state, I will advocate not allowing
REMOVE of OPEN files.

	-David

---

From: Mike Eisler <[email-masked]>
Date: Wed, 26 Sep 2001 12:05:23 -0700

> accomplish with the string of COMPOUND operations.

I have to agree. For example it is useful to distinguish a 
sequence of operations that has LOOKUP from the many
possible uses, such a mounting a filesystem,
stat()ing a file, etc.