Subject: [nfsv4] Proposed resolutions for some pnfs-related [[Comment]] sections

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 2 Nov 2007 14:49:29 -0400

I'm trying to get rid of many [[Comment]] sections and there are a 
few discussed below that I'd like to run by people.  Let me know
right away if you have an issue with any of the proposed resolutions.

The first concerns the possibility which might exist for a file 
layout type to have a COMPOUND that contains some ops addressing
the server as a metadata server and others as a data server.  I
think we should disallow this.

Here's what is in nfsv41_middle_file_layout.xml right now.

 <t>
  Clients MUST use the filehandle described within the
  layout when accessing data on NFSv4.1 data servers. When
  using the layout's filehandle, the client MUST only
  issue the NULL procedure and the COMPOUND procedure's
  BACKCHANNEL_CTL, BIND_CONN_TO_SESSION, CREATE_SESSION,
  COMMIT, DESTROY_CLIENTID, DESTROY_SESSION, EXCHANGE_ID,
  READ, WRITE, PUTFH, SECINFO_NO_NAME, SET_SSV, and
  SEQUENCE operations to the NFSv4.1 data server associated
  with that data server filehandle.  If a client issues an
  operation to the data server other than those specified
  above, using the filehandle and data server listed
  in the file's layout, that data server MUST return
  NFS4ERR_NOTSUPP to the client, unless the server's
  EXCHANGE_ID results returned (EXCHGID4_FLAG_USE_PNFS_DS |
  EXCHGID4_FLAG_USE_PNFS_MDS) or (EXCHGID4_FLAG_USE_PNFS_DS
  | EXCHGID4_FLAG_USE_NON_PNFS), see <xref
  target="pnfs_session_stuff"/> As described in <xref
  target="layout_semantics" />, a client MUST NOT issue
  an I/O to a data server for which it does not hold a
  valid layout; the data server MUST reject such an I/O.

  <cref>

  We should discuss mixing of MDS and DS operations in the
  same compound, but need to find other places where this
  is discussed and move that discussion here.

  </cref>
 </t>

I don't see those other places and I'd like to replace the
paragraphs above with:	

 <t>
  Clients accessing data on an NFSv4.1 data server MUST issue
  only the NULL procedure and COMPOUND procedures whose
  operations are taken only from two restricted
  subsets of the operations defined as valid NFSv4.1 
  operations.  Clients MUST use the filehandle contained 
  within the layout when accessing data on NFSv4.1 data 
  servers. 
 </t>
 <t>
  The first of these operation subsets consist of
  session management operations where the current
  filehandle is not relevant.  This subset consists 
  of the BACKCHANNEL_CTL, BIND_CONN_TO_SESSION, CREATE_SESSION,
  DESTROY_CLIENTID, DESTROY_SESSION, EXCHANGE_ID,
  SECINFO_NO_NAME, SET_SSV, and SEQUENCE operations.
  The client may use these operations in order to set
  up and maintain the appropriate client IDs and
  sessions involved in communication with the data
  server.
 </t>
 <t>
  The second subset consists of COMMIT, READ, WRITE, PUTFH, 
  and SECINFO_NO_NAME.  These operations must be
  used with a current filehandle taken from the 
  layout.  In the case of PUTFH, the new current
  filehandle must be one taken from the layout.
  As described in <xref
  target="layout_semantics" />, a client MUST NOT issue
  an I/O to a data server for which it does not hold a
  valid layout; the data server MUST reject such an I/O.
 </t>
 <t>
  Unless the server has a concurrent non-data-server
  personality, i.e. EXCHANGE_ID results returned 
  (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_PNFS_MDS) 
  or (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS), 
  see <xref target="pnfs_session_stuff"/>, any use of
  operations other than those specified in the two 
  subsets above MUST return
  NFS4ERR_NOTSUPP to the client.
 </t>
 <t>
  When the server has concurrent data server and 
  non-data-server personalities, each COMPOUND operation
  is one of:
  <list style="symbols'>
    <t>
      A COMPOUND devoted to session/client housekeeping,
      which applies without distinction to both personalities.
    </t>
    <t>
      A COMPOUND which is referable to the data server
      personality.
    </t>
    <t>
      A COMPOUND which is referable to the non-data-server
      personality.
    </t>
  </list>
 </t>
 <t>
  It is not allowed for a single COMPOUND to contain operations
  acting on both individual personalities, although an COMPOUND
  pertaining to a single individual personality may contain
  housekeeping operation which affect both personalities.
 </t>
 <t>
  When a COMPOUND first executes an operation which is not
  part of one of the two sets above, or when it executes a
  PUTFH containing a filehandle not derived from a layout,
  it acts as a normal COMPOUND on any other server and the 
  data server personality ceases to be relevant. 
  There are no special restrictions on the 
  operations in the COMPOUND to limit then to those for
  a data server.  When a PUTFH is done, filehandles
  derived from the layout are not valid.  If their format
  is not normally acceptable, then NFS4ERR_BADHANDLE MUST
  result.  Similarly, current filehandles for other operations
  do not accept filehandles derived from layouts and these
  will result in NFS4ERR_STALE. 
 </t>
 <t>
  When a COMPOUND first executes a PUTFH where the filehandle
  is one from a layout, the COMPOUND henceforth is interpreted
  with respect to the data server personality.  In
  particular, operations outside the two classes discussed
  above MUST result in NFS4ERR_NOTSUPP and filehandles
  are validated using the rules of the data server,
  resulting in NFS4ERR_BADHANDLE and/or NFS4ERR_STALE
  even when they would not normally do so when addressed
  to the non-data-server personality.
 </t>	

The next item relates to stateids and IO mode validations:

 <section title="Open and Deny Mode Validation">
 <t>
  Open and deny mode validation MUST be performed against
  the open and deny mode(s) held by the data servers. When
  access is reduced or a deny mode made more restrictive
  (because of CLOSE or DOWNGRADE) the data server MUST
  prevent any I/Os that would be denied if performed on the
  metadata server. Conversely, when access is expanded,
  the data server MUST NOT reject a request because of
  open or deny issues without making sure that the open
  mode upgrade or deny mode relaxation is not in progress.
  <cref>
  It seems like the recent (June/July 2007)
  stateid.seqid discussion might modify this;
  a client might send a new state.seqid, that tells the data server
  it's seqid is out of date; the data server might return
  NFS4ERR_DELAY.
  </cref>

 </t>
 </section>	

In addition to the cref section there seems to me to be confusion
in the other parts of this and adjoining sections.  The big issue
is that there is discussion of getting stateid seqids right but
that seems to me pointless complexity for the data server.  Given
parallel open the general pattern will be to use seqid=0 and only
use non-zero seqid for a few special operation like OPEN_DOWNGRADE
and CLOSE, none of which is going to be sent to the data server.

So I'm proposing that the stateids for the data server always have
seqid as zero and you get BAD_STATEID if not.

Another thing that is wrong about this section is the discussion of
deny modes.  Deny modes have their effect at open time, and, unless
we have IO's with special stateids (which I believe are not allowed) 
do not affect IO's.

Also, "without making sure that the open mode upgrade or deny mode 
relaxation is not in progress" seems flat-out wrong.  You can never 
be assured that it is not in progress when you actually do the
rejection.
If you check that it is not an a given instant and the proceed to
do the rejection, it might be in progress when you actually do the
rejection.  All you should check is that it hasn't previously been
made allowable.

We have the following issue with access time in LAYOUTCOMMIT:

    <t>
      The loca_time_modify and loca_time_access <cref>If LAYOUTCOMMIT
      is only for writes, then why update access time?</cref> fields
      allow the client to suggest times it would like the metadata
      server to set.  The metadata server may use these time values or
      it may use the time of the LAYOUTCOMMIT operation to set these
      time values.  If the metadata server uses the client provided
      times, it should ensure time does not flow backwards.  If the
      client wants to force the metadata server to set an exact time,
      the client should use a SETATTR operation in a compound right
      after LAYOUTCOMMIT.  See <xref target="committing_layout" /> for
      more details.  If the new client desires the resultant mtime or
      atime, it should construct the COMPOUND so that a GETATTR
      follows the LAYOUTCOMMIT.
    </t>

The issue of loca_access_time was discussed at the pnfs review but as
I understand things, there was no resolution that got universal
approval.  The simplest thing, which I'm going to propose, if nobody
has a better idea, is simply to get of rid loca_access_time.

Also we have the following inconsistent paragraphs:

    <t>
      When lr_returntype is LAYOUTRETURN4_FILE, the layout being
      returned may be a subdivision of a layout previously obtained
      with LAYOUTGET.  The layout being returned may also be a subset
      or superset of a layout specified by CB_LAYOUTRECALL.  However,
      if it is a subset, the recall is not complete until the full
      recalled scope has been returned.  Recalled scope refers to the
      octet range in the case of LAYOUTRETURN4_FILE, use of
      LAYOUTRETURN4_FSID, or the use of LAYOUTRETURN4_ALL.  It
      is also permissible, and no error should result, for a client to
      return a octet range covering a layout it does not hold.  If the
      lrf_length is all 1s, the layout covers the range from
      lrf_offset to EOF.
    </t>

And,

    <t>
      If the layout identified in the arguments does not exist, the
error
      NFS4ERR_BADLAYOUT is returned.  If a layout exists, but the iomode
      does not match, NFS4ERR_BADIOMODE is returned. 
      <cref> 
	This error
	return description does not align with the statement above that
	the client should not receive an error when it returns a layout
	that it does not hold. 
      </cref>
    </t>
		
It appears to me that the intent of the first paragraph cited was to 
simplify the job of the client, particularly in those cases (like the
file layout type) where layouts are simple so that it can avoid detailed
logic to keep precise track of the octet ranges for which it has a
layout.  
On the other hand, having the server be unable to report cases in which 
a the client has lost its mind is troublesome.  Here's my suggested 
resolution.

    <t>
      When lr_returntype is LAYOUTRETURN4_FILE, the layout being
      returned may be a subdivision of a layout previously obtained
      with LAYOUTGET.  The layout being returned may also be a subset
      or superset of a layout specified by CB_LAYOUTRECALL.  However,
      if it is a subset, the recall is not complete until the full
      recalled scope has been returned.  Recalled scope refers to the
      octet range in the case of LAYOUTRETURN4_FILE, use of
      LAYOUTRETURN4_FSID, or the use of LAYOUTRETURN4_ALL.  It
      is also permissible, and no error should result, for a client to
      return an octet range covering a layout it does not currently
hold,
      as long as it, at some time held a layout for some octet range of
      file in question. If the lrf_length is all 1s, the layout covers 
      the range from lrf_offset to the last octet for which the client 
      actually holds a layout.
    </t>

And,

    <t>
      If it determinable that layout identified in the argument is such
      that the client could never have possessed a layout for that file
      (of any octet range) NFS4ERR_BADLAYOUT should be returned.  If a 
      layout exists, but the iomode does not match, NFS4ERR_BADIOMODE 
      is returned. 
    </t>

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Benny Halevy <[email-masked]>
Date: Sun, 04 Nov 2007 08:59:44 +0200

> think we should disallow this.

<snip>

>   housekeeping operation which affect both personalities.

This makes sense to me.

<snip>

> seqid as zero and you get BAD_STATEID if not.

I thought that the client should just use the most recent stateid returned
to it by the MDS, as per 14.10.1:
"When the client issues I/O to a data server, the stateid used must be one previously returned by the metadata server. Permitted stateids include an open stateid (the stateid field of data type OPEN4resok as returned by OPEN), a delegation stateid (the stateid field of data types open_read_delegation4 and open_write_delegation4 as returned by OPEN or WANT_DELEGATION, or as sent by CB_PUSH_DELEG), or a stateid returned by the LOCK or LOCKU operations."

> we have IO's with special stateids (which I believe are not allowed) 

Right

> has a better idea, is simply to get of rid loca_access_time.

I object to doing that.
Since the MDS does intercept the I/Os going to the DSs loca_access_time
will be useful to the MD server, otherwise, if it supports atime
(and is configured to track it), it will be forced to get the atime
from all the file's DSs which can have a significant hit on performance.
Note that unlike an explicit SETATTR this value is basically a hint and
the server does not have to respect it.

Also, I would like the spec to be clear about allowing LAYOUTCOMMIT
for reads as this is still needed for the objects layout type for reporting
read errors.

>       layout exists, but the iomode does not match, NFS4ERR_BADIOMODE 

I'm OK with that.

---

From: Robert Gordon <[email-masked]>
Date: Sun, 04 Nov 2007 10:48:28 -0600

>   within the layout when accessing data on NFSv4.1 data

strictly speaking the last sentence is not true; since
the filehandle may be the MDS filehandle, and that is
not conveyed in the layout.

Perhaps:

"Clients MUST use the correct filehandle defined by
  the layout when accessing data on NFSv4.1 data
  servers. "

Robert.

---

From: Robert Gordon <[email-masked]>
Date: Sun, 04 Nov 2007 11:19:52 -0600

> to the non-data-server personality.

"It is not allowed" -- should that be a SHOULD NOT or
MUST NOT ?

Since the filehandle could be the MDS filehandle; I'm not
sure that the server can distinguish which personality
the PUTFH pertains to.. But maybe that is your point ?
-- if it is then we need to be clearer :) -- 

Robert.

---

From: Robert Gordon <[email-masked]>
Date: Sun, 04 Nov 2007 12:25:35 -0600

On Nov 4, 2007, at 1:59 AM, Benny Halevy wrote:

> "When the client issues I/O to a data server, the stateid used must  
> be one
> previously returned by the metadata server. Permitted stateids  
> include an
> open stateid (the stateid field of data type OPEN4resok as returned  
> by OPEN),
> a delegation stateid (the stateid field of data types  
> open_read_delegation4
> and open_write_delegation4 as returned by OPEN or WANT_DELEGATION,  
> or as
> sent by CB_PUSH_DELEG), or a stateid returned by the LOCK or LOCKU  
> operations."

I think that Dave is going to update that text.. or in the process of..

I think using zero in the stateid sequence field is okay.

We also need to say that the client should use either the OPEN or
DELEGATION stateid, and may use the lock stateid (with the most recent
sequence id) if mandatory locking is implemented.

Robert.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 4 Nov 2007 19:10:43 -0500

> strictly speaking the last sentence is not true; 

Why not?

> since the filehandle may be the MDS filehandle, 

Where does the current spec say that?

> and that is not conveyed in the layout.

Probably not.

In draft-15, the first sentence of section 14.7 is:

   Clients MUST use the filehandle described within the layout when
   accessing data on NFSv4.1 data servers. 

That doesn't sound to me the like the filehandle may be the MDS
filehandle.

My intention was not change the spec in this regard.  Is there some
other text that allows this?

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Sunday, November 04, 2007 11:48 AM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some pnfs-related
[[Comment]]sections

>   within the layout when accessing data on NFSv4.1 data

strictly speaking the last sentence is not true; since the filehandle
may be the MDS filehandle, and that is not conveyed in the layout.

Perhaps:

Robert.

---

From: "Noveck, Dave" <[email-masked]>
Date: Sun, 4 Nov 2007 19:22:09 -0500

> > It is not allowed for a single COMPOUND to contain operations acting

> > on both individual personalities, although an COMPOUND pertaining to
a 
> > single individual personality may contain housekeeping operation
which 
> > affect both personalities.

> "It is not allowed" -- should that be a SHOULD NOT or MUST NOT ?

It is mandatory but simply putting the words "MUST NOT" in there doesn't
result in a correct statement.  The issue is that the clinet can create
any fool COMPOUND he wants but the server is obliged not to give the
client what he wants.  His MDS operations after the DS operation in the
same COMPOUND will get NOTSUPP (the server MUST give him that).
Similarly with DS operations after MDS operations.  The server will not
recognize DS filehandles specially.

How about?

A server MUST NOT execute operations within a single COMPOUND acting on
behalf of both individual personalities, although the execution of a
COMPOUND pertaining to a single individual personality may contain
houskeeping operations affecting both personalities.

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Sunday, November 04, 2007 12:20 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some pnfs-related
[[Comment]]sections

On Nov 2, 2007, at 1:49 PM, Noveck, Dave wrote:
> <t>
> It is not allowed for a single COMPOUND to contain operations acting 
> on both individual personalities, although an COMPOUND pertaining to a

> single individual personality may contain housekeeping operation which

> affect both personalities.
> </t>
> <t>
> When a COMPOUND first executes an operation which is not part of one 
> of the two sets above, or when it executes a PUTFH containing a 
> filehandle not derived from a layout, it acts as a normal COMPOUND on 
> any other server and the data server personality ceases to be 
> relevant.
> There are no special restrictions on the operations in the COMPOUND to

> limit then to those for a data server.  When a PUTFH is done, 
> filehandles derived from the layout are not valid.  If their format is

> not normally acceptable, then NFS4ERR_BADHANDLE MUST result.  
> Similarly, current filehandles for other operations do not accept 
> filehandles derived from layouts and these will result in 
> NFS4ERR_STALE.
> </t>
> <t>
> When a COMPOUND first executes a PUTFH where the filehandle is one 
> from a layout, the COMPOUND henceforth is interpreted with respect to 
> the data server personality.  In particular, operations outside the 
> two classes discussed above MUST result in NFS4ERR_NOTSUPP and 
> filehandles are validated using the rules of the data server, 
> resulting in NFS4ERR_BADHANDLE and/or NFS4ERR_STALE even when they 
> would not normally do so when addressed to the non-data-server 
> personality.
> </t>	

"It is not allowed" -- should that be a SHOULD NOT or MUST NOT ?

Since the filehandle could be the MDS filehandle; I'm not sure that the
server can distinguish which personality the PUTFH pertains to.. But
maybe that is your point ?
-- if it is then we need to be clearer :) -- 

Robert.

---

From: Robert Gordon <[email-masked]>
Date: Sun, 04 Nov 2007 22:31:53 -0600

> other text that allows this?

14.3 File Layout Data Types.

[snip]

	4. nfl_fh_list. An array of data server filehandles for each list of  
data
	servers in each element of the nflda_multipath_ds_list array. The  
number
	of elements in nfl_fh_list depends on whether sparse or dense  
packing is
	being used. If sparse packing is being used, the number of elements in
	nfl_fh_list MUST be of three values:
		Zero. This means that filehandles used for each data server are
		the same as the filehandle returned by the OPEN operation from
		the metadata server.
[snip]

Robert..

---

From: Robert Gordon <[email-masked]>
Date: Sun, 04 Nov 2007 23:15:39 -0600

On Nov 4, 2007, at 6:22 PM, Noveck, Dave wrote:

> housekeeping operations affecting both personalities.
>

I think that is fine, however an MDS may be claiming both personalities
and also handing out layouts such that the client has to use the
FH returned by OPEN.. as silly as that may seem. :)

Robert.

---

From: "Muntz, Daniel" <[email-masked]>
Date: Mon, 5 Nov 2007 11:30:33 -0800

Is anyone using the special case of 14.3?

If the spcial case is removed, and a particular implementation wants to
use the MDS FH as the DS FH, it may do so.  It can call the MDS FH a DS
FH for one or more of the DSs, and the spec would be agnostic on the
issue (i.e., you're not using an MDS FH in a layout; you're using a DS
FH that happens to be the same as the MDS FH).

In my (possibly oversimplified) mind, the MDS FH is "the" FH for the
file, to be used for meta-data ops, etc.  The DS FHs are part of the
mechanism for accessing stripes of the file.  Is there a compelling
reason to make the client deal with distinguishing between whether it's
going to use an MDS FH vs. a DS FH for a stripe operation?  Is there a
case where it makes sense to complicate the meaning of "MDS FH" vs. "DS
FH" by allowing this special case?

  -Dan

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Sunday, November 04, 2007 8:32 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some
pnfs-related[[Comment]]sections

[snip]

	4. nfl_fh_list. An array of data server filehandles for each
list of data
	servers in each element of the nflda_multipath_ds_list array.
The number
	of elements in nfl_fh_list depends on whether sparse or dense
packing is
	being used. If sparse packing is being used, the number of
elements in
	nfl_fh_list MUST be of three values:
		Zero. This means that filehandles used for each data
server are
		the same as the filehandle returned by the OPEN
operation from
		the metadata server.
[snip]

Robert..

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 5 Nov 2007 16:35:42 -0500

> If the special case is removed, and a particular implementation wants
to 
> use the MDS FH as the DS FH, it may do so.

But if you allow it do so, then, even though the encouragement of the
special case is gone, you are still allowing a situation in which it is 
ambiguous for a given op wherther it is being executed by the MDS or
the DS.  Although we can write text allow for that, it strikes me as
something we will wind up regrettting.

> Is there a compelling reason to make the client deal with
distinguishing 
> between whether it's going to use an MDS FH vs. a DS FH for a stripe 
> operation? 

I don't think this is a client issue.  The client needs to distiguish 
whether he is talking to MDS and DS and that is going to be true
whatever we do about this issue.

> Is there a case where it makes sense to complicate the meaning of 
> "MDS FH" vs. "DS FH" by allowing this special case?

It's the server that has the complexity when he receives an FH and
has to distinguish which one it is (when it may not be disguishable),
and thus which personality a given oepration is being directed at.

-----Original Message-----
From: Muntz, Daniel 
Sent: Monday, November 05, 2007 2:31 PM
To: Robert Gordon; Noveck, Dave
Cc: [email-masked]
Subject: RE: [nfsv4] Proposed resolutions for some

  -Dan

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Sunday, November 04, 2007 8:32 PM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some

[snip]

Robert..

---

From: Robert Gordon <[email-masked]>
Date: Mon, 05 Nov 2007 15:53:54 -0600

> and thus which personality a given oepration is being directed at.

A server could be implemented in such a way that the stateid conveys
the fact that the IO is intended for the data-server persona...  
couldn't it ?

Robert..

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 5 Nov 2007 17:02:24 -0500

Yes it could.  In fact, there are reasons that they pretty well
have to be distinguishable.  An MDS-based stateid can reply on the
cleintid (gotten from the sessionid) for uniqueness there the DS
stateids which are going to be interpreted by DS's with other
sessions don't have that ability.

I think I'm going to try to come with attempt-2 to deal with this
issue.  One complexity is that there are some DS ops that don't have 
stateids.

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Monday, November 05, 2007 4:54 PM
To: Noveck, Dave
Cc: Muntz, Daniel; [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some

Robert..

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 6 Nov 2007 15:12:26 -0500

> I think that Dave is going to update that text.. or in the process
of..

I've made some updates but I want to make sure we have consensus here
and 
so I can make whatver additional changes are required and close this
out.

> I think using zero in the stateid sequence field is okay.

OK, but I'm saying that for the DS non-zero stateid sequence field 
should be not OK.

All of the reasons for allowing this on the MDS have to do with parallel
opens/closes and needing ways to prevent the effects of races.  It is 
allowed on other ops that take a stateid for reasons of symmetry.

In the case of a DS, it is just simpler to say that seqid field zero
is the only allowable case.  That would avoid making implementation
to work to support somethng that doesn't seem to have any value.

> sequence id) if mandatory locking is implemented.

How does the client know if mandatory locking is implemented?

As to the seqid, I don't see the value.  Since the DS cannot be sure
that you are sending the correct one, it needs to validate it assuming
you are not.  But if it capable of doing that, then it should be capable
of getting the appropriate state to use, without you sending the seqid.

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Sunday, November 04, 2007 1:26 PM
To: Benny Halevy; Noveck, Dave
Cc: NFSv4
Subject: Re: [nfsv4] Proposed resolutions for some pnfs-related
[[Comment]] sections

Robert.

---

From: Robert Gordon <[email-masked]>
Date: Tue, 06 Nov 2007 14:44:36 -0600

> to work to support somethng that doesn't seem to have any value.

I agree; The seqid MUST be zero. If not then the DS will return
NFS4ERR_INVAL (?)

> How does the client know if mandatory locking is implemented?

NFS4ERR_LOCKED would have been returned to the client if it had used
an open or delegation stateid;

> of getting the appropriate state to use, without you sending the  

You are right; and I agree.

I'd forgotten the following :-

"14.10.2.1.  Lock State Propagation

If the pNFS server supports mandatory locking, any
mandatory locks on a file MUST be made effective at
the data servers before the request that establishes
them returns to the caller. Thus, mandatory lock state
MUST be synchronously propagated to the data servers."

and (for completeness) we also say this in the last
paragraph of 13.5.1:

"In the presence of pNFS functionality, mandatory file
locks MUST behave as they would without pNFS. Therefore,
if mandatory file locks and layouts are provided simultaneously,
the storage device MUST be able to enforce the mandatory
file locks. For example, if one client obtains a mandatory
lock and a second client accesses the storage device, the
storage device MUST appropriately restrict I/O for the byte
range of the mandatory file lock. If the storage device is
incapable of providing this check in the presence of mandatory
file locks, the metadata server then MUST NOT grant layouts
and mandatory file locks simultaneously."

--
Robert.

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 06 Nov 2007 14:50:28 -0600

> to work to support somethng that doesn't seem to have any value.

I agree with this.  The stateid sent in requests destined for at data
server MUST have the seqid portion == 0.

> How does the client know if mandatory locking is implemented?

That is the fundamental issue then, right?  Without a method
of determining if mandatory file locking is being used at
the server, the client's behavior can not vary.  Either the
client is required to use the lock stateid (if file locking
is being used by the client) or it isn't required to do so.

> of getting the appropriate state to use, without you sending the  

If the pNFS server is providing mandatory file locking then it
would presumably have a method of determining the client's
access to the file.  The seqid is a method of determining at
what point in the state transitions the client was "located"
when submitting the I/O.

So another way to phrase your response to this is that the
we, as protocol designers, are not concerned with the timeline
of access (did the client submit an I/O and simultaneously
submit a LOCK/LOCKU operation that changes its access).
Therefore, in the case of mandatory file locking support, the
data server is only required to know about changes in the
file locking state such that it can confirm the client's access
with respect to the current file locks held and not the
past state of file locks held.

In summary, the seqid of the stateid supplied to the data server
is required to be zero regardless of stateid type.

We then decide if a lock stateid is required in the event that
the server is supporting mandatory file locking.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 6 Nov 2007 15:53:58 -0500

> I agree; The seqid MUST be zero. If not then the DS will return

I was thinking BAD_STATEID since every op that can receive this already
has to have BAD_STATEID in its error list.  I'm trying to avoid
requiring any more changes to the error/op lists, although I hope I
won't push that to exteremes.

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Tuesday, November 06, 2007 3:45 PM
To: Noveck, Dave
Cc: Benny Halevy; NFSv4
Subject: Re: [nfsv4] Proposed resolutions for some pnfs-related
[[Comment]] sections

--
Robert.

---

From: Robert Gordon <[email-masked]>
Date: Tue, 06 Nov 2007 15:05:55 -0600

> won't push that to exteremes.

I agree.

(but think that NAUGHTY_STATEID would add some humor to
the protocol)

Robert.

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 6 Nov 2007 18:15:17 -0500

> NFS4ERR_LOCKED would have been returned to the client if it had 
> used an open or delegation stateid; 

If there is an effective mandatory lock that would prevent the IO
without us knowing the lockowner that we get LOCKED and he client will
subsequently, for that open file, use the locking stateid if one is
available.  I think that works OK.  If he hasn't got LOCKED he SHOULD
NOT use the lock stateid.

If he has a delegation, then the delegation stateid should be used.  In
that case, the client has an assurance that either all opens are from
that one client (write delegation case) and the client is capable of
checking himself among the various lockowners or there can be no
mandatory locks (ead delegation case) and IO should just go through. 

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Tuesday, November 06, 2007 3:45 PM
To: Noveck, Dave
Cc: Benny Halevy; NFSv4
Subject: Re: [nfsv4] Proposed resolutions for some pnfs-related
[[Comment]] sections

On Nov 6, 2007, at 2:12 PM, Noveck, Dave wrote:

> is the only allowable case.  That would avoid making implementation to

> work to support somethng that doesn't seem to have any value.

>> DELEGATION stateid, and may use the lock stateid (with the most 
>> recent sequence id) if mandatory locking is implemented.
>

NFS4ERR_LOCKED would have been returned to the client if it had used an
open or delegation stateid;

> you are not.  But if it capable of doing that, then it should be 
> capable of getting the appropriate state to use, without you sending 
> the seqid.

If the pNFS server supports mandatory locking, any mandatory locks on a
file MUST be made effective at the data servers before the request that
establishes them returns to the caller. Thus, mandatory lock state MUST
be synchronously propagated to the data servers."

and (for completeness) we also say this in the last paragraph of 13.5.1:

"In the presence of pNFS functionality, mandatory file locks MUST behave
as they would without pNFS. Therefore, if mandatory file locks and
layouts are provided simultaneously, the storage device MUST be able to
enforce the mandatory file locks. For example, if one client obtains a
mandatory lock and a second client accesses the storage device, the
storage device MUST appropriately restrict I/O for the byte range of the
mandatory file lock. If the storage device is incapable of providing
this check in the presence of mandatory file locks, the metadata server
then MUST NOT grant layouts and mandatory file locks simultaneously."

--
Robert.

---

From: Robert Gordon <[email-masked]>
Date: Tue, 06 Nov 2007 17:33:09 -0600

> NOT use the lock stateid.

Correct, that is what i was thinking.

> mandatory locks (ead delegation case) and IO should just go through.

yes..

--
Robert...
"Don't Make Assumptions"

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 7 Nov 2007 13:08:51 -0500

> > has a better idea, is simply to get of rid loca_access_time.

OK, but right now, we have a situation in LAYOUTCOMMIT is discussed in
several dozen places, and most them implicitly give the impression and
many of them explicitly (including the description of LAYOUTCOMMIT
itself) say that this is to be used when writing.  That seems to be 
inconsistent with having access time be a field in that and that is the
issue that is raised in the <cref>, and that we have to address very 
soon.

> from all the file's DSs which can have a significant hit on
performance.

Asking the clients to communicate with the MDS to propagate this will
also have a performance cost.  It is true that the client would have to
update only a single MDS, but if this is within the control protocol, we
have opportunities for extensive batching of updates to multiple files 
to be transferred at the same time, so it is not clear to me that making
the client do this is a net win.

> for reads as this is still needed for the objects layout type for
reporting

This is not a case in which the spec is unclear and you can simply 
clarify it allowing LAYOUTCOMMIT on read.  It seems to me that you are 
asking for a fundamentally different treatment of LAYOUTCOMMIT making it

something that can/should/must (depending on what?) be used for various
types of layouts under various conditions.  That is very different from 
what is there now and I don't see any way that that level of rewriting 
can be done in time to get it into draft-16.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Sunday, November 04, 2007 2:00 AM
To: Noveck, Dave
Cc: [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some pnfs-related
[[Comment]] sections

<snip>

This makes sense to me.

<snip>

> seqid as zero and you get BAD_STATEID if not.

I thought that the client should just use the most recent stateid
returned
to it by the MDS, as per 14.10.1:
"When the client issues I/O to a data server, the stateid used must be
one previously returned by the metadata server. Permitted stateids
include an open stateid (the stateid field of data type OPEN4resok as
returned by OPEN), a delegation stateid (the stateid field of data types
open_read_delegation4 and open_write_delegation4 as returned by OPEN or
WANT_DELEGATION, or as sent by CB_PUSH_DELEG), or a stateid returned by
the LOCK or LOCKU operations."

Right

I object to doing that.

Also, I would like the spec to be clear about allowing LAYOUTCOMMIT
for reads as this is still needed for the objects layout type for
reporting
read errors.

I'm OK with that.

---

From: Benny Halevy <[email-masked]>
Date: Wed, 07 Nov 2007 20:31:53 +0200

Can we optionally provide access_time and/or a layout_type specific structure
piggybacked on LAYOUTRETURN instead?

Benny

-- 
Benny Halevy
Software Architect
Tel/Fax: +972-3-647-8340
Mobile: +972-54-802-8340
US:      +1-412-203-3187
[email-masked]
 
Panasas, Inc.
Leader in Parallel Storage
www.panasas.com

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 07 Nov 2007 12:40:05 -0600

If the only purpose of the access time field in layoutcommit is to
allow the client the ability to update it (no release of allocations
or other side -effects), wouldn't SETATTR be sufficient for the client?
Or is there concern about appropriate access in setting that attribute
where a layout based update of access time would be better?

Also, given that many NFS servers disable access time updates for
performance reasons, is this a hard/strict requirement?

Spencer

On Nov 7, 2007, at 12:31 PM, Benny Halevy wrote:

>>>       after LAYOUTCOMMIT.  See <xref target="committing_layout" / 
>>> > for

---

From: Benny Halevy <[email-masked]>
Date: Wed, 07 Nov 2007 21:03:08 +0200

> where a layout based update of access time would be better?

explicit setattr won't work the same way if multiple clients are
reading the file concurrently as it may set the access time backwards.
The intention of the LAYOUTCOMMIT hint was to only set the access_time
on the file if the client's reported access_time is later than the
current access_time (and not in the future in which case the current time
should be used)

> performance reasons, is this a hard/strict requirement?

We have customers that demand atime support so I would say that
the ability to maintain it as efficiently as possible is important.
Blocks can't get the atime from the LUNs and for objects
it's impossible to amortize the cost of getting the atime
from the OSDs over multiple objects in a batch. So doing
the "soft" update at LAYOUTCOMMIT or LAYOUTRETURN time is
essential.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Wed, 7 Nov 2007 11:13:52 -0800 (PST)

> reading the file concurrently as it may set the access time

                  enum time_how4 {
                          SET_TO_SERVER_TIME4 = 0,
                          SET_TO_CLIENT_TIME4 = 1
                  };

   settime4

                  union settime4 switch (time_how4 set_it) {
                   case SET_TO_CLIENT_TIME4:
                           nfstime4       time;
                   default:
                           void;
                  };

The client can set set_it to SET_TO_SERVER_TIME4 when
it issues the SETATTR.

As I recall saying in the pNFS review, LAYOUTCOMMIT seems to
be replicating functionality we already have.

> current access_time (and not in the future in which case the current

That seems over engineered. Accurate access_time has never
been something NFS (and probably every other networked file system)
could guarantee.

> We have customers that demand atime support so I would say that

What precisely are they demanding?

That every read() from the client's cache update the atime?

I get that all the time from customers and QA engineers, and
I tell them they can use direct I/O to the server if that's
what they think they need. 

And I would tell the customer that perfectly accurate atime is
something that won't be provided with pNFS. We eliminated
GETATTR from the set of operations a file-based data server can
support for a reason.

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 07 Nov 2007 13:30:35 -0600

> be replicating functionality we already have.

The client may also use NVERIFY if there is a specific value
that it desires to compare against.

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Thu, 08 Nov 2007 17:23:22 +0200

> The client can set set_it to SET_TO_SERVER_TIME4 when

OK. Then that leaves us only with our requirement for 
layout-type specific payload in LAYOUTRETURN.
There shouldn't be any problem with adding that, right?

> GETATTR from the set of operations a file-based data server can

I can't be specific about the requirements but they don't require a
high level of accuracy.

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 8 Nov 2007 11:58:29 -0500

> > Yes it could.

Actually it can't.  This is a consequence of the global stateid model
for 
the file layout.  Sigh!

So, given that, here is my second attempt at addressing this:

 <t>
  Clients accessing data on an NFSv4.1 data server MUST issue
  only the NULL procedure and COMPOUND procedures whose
  operations are taken only from two restricted
  subsets of the operations defined as valid NFSv4.1 
  operations.  Clients MUST use the filehandle contained 
  within the layout when accessing data on NFSv4.1 data 
  servers. 
 </t>
 <t>
  The first of these operation subsets consist of
  session management operations where the current
  filehandle is not relevant.  This subset consists 
  of the BACKCHANNEL_CTL, BIND_CONN_TO_SESSION, CREATE_SESSION,
  DESTROY_CLIENTID, DESTROY_SESSION, EXCHANGE_ID,
  SECINFO_NO_NAME, SET_SSV, and SEQUENCE operations.
  The client may use these operations in order to set
  up and maintain the appropriate client IDs and
  sessions involved in communication with the data
  server.  Henceforth these will be referred to as
  data-server housekeeping operations.
 </t>
 <t>
  The second subset consists of COMMIT, READ, WRITE, PUTFH, 
  and SECINFO_NO_NAME.  These operations must be
  used with a current filehandle taken from the 
  layout.  In the case of PUTFH, the new current
  filehandle must be one taken from the layout.
  Henceforth, these will be referred to as data-server
  IO operatons.
  As described in <xref target="layout_semantics" />, 
  a client MUST NOT issue
  an I/O to a data server for which it does not hold a
  valid layout; the data server MUST reject such an I/O.
 </t>
 <t>
  Unless the server has a concurrent non-data-server
  personality, i.e. EXCHANGE_ID results returned 
  (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_PNFS_MDS) 
  or (EXCHGID4_FLAG_USE_PNFS_DS | EXCHGID4_FLAG_USE_NON_PNFS), 
  see <xref target="pnfs_session_stuff"/>, any use of
  operations other than those specified in the two 
  subsets above MUST return
  NFS4ERR_NOTSUPP to the client.
 </t>
 <t>
  When the server has concurrent data server and 
  non-data-server personalities, each COMPOUND sent by the 
  client MUST be constructed
  so that it appropriate to one of the two personalities, and
  must not contain operations directed to a mix of those 
  personalities.  The server MUST enforce this.  To understand
  the constraints, operations within a COMPOUND are divided into
  the following three classes: 
  <list style="numbers">
    <t>
      An operation which is ambiguous regarding its personality
      assignment.  These include all of the data-server
      housekeeping operations.  Additionally, if the
      server has assigned filehandles so that the ones defined
      by the layout are the same as those used by the meta-data 
      server, all operations in the second class are within this
      group unless a stateid used is incompatible with a
      data-server personality in that it is a special stateid
      or has a non-zero seqid field.
    </t>
    <t>
      An operation which is referable to the data server
      personality.  These are data-server IO operations
      where the filehandle is one
      that can only be validly directed to the data-server
      personality.
    </t>
    <t>
      An operation which is referable to the non-data-server
      personality.  These include all COMPOUND operations that are
      neither data-server housekeeping nor data-server IO 
      operations plus data-server IO operations where the 
      current fh (or the one to be made the current fh in the
      case of PUTFH) is one that is only valid on the metadata
      server or where a stateid is used that is incompatible 
      with the data server, i.e. is a spcial stateid or has
      a non-zero seqid value.
    </t>
  </list>
 </t>
 <t>
  When a COMPOUND first executes an operation from class 3 above,
  it acts as a normal COMPOUND on any other server and the 
  data server personality ceases to be relevant. 
  There are no special restrictions on the 
  operations in the COMPOUND to limit then to those for
  a data server.  When a PUTFH is done, filehandles
  derived from the layout are not valid.  If their format
  is not normally acceptable, then NFS4ERR_BADHANDLE MUST
  result.  Similarly, current filehandles for other operations
  do not accept filehandles derived from layouts and not 
  normally usable on metadata-server and using these
  will result in NFS4ERR_STALE. 
 </t>
 <t>
  When a COMPOUND first executes an operation from class 2,
  which would be PUTFH where the filehandle
  is one from a layout, the COMPOUND henceforth is interpreted
  with respect to the data server personality. 
  Operations outside the two classes discussed
  above MUST result in NFS4ERR_NOTSUPP.  Filehandles
  are validated using the rules of the data server,
  resulting in NFS4ERR_BADHANDLE and/or NFS4ERR_STALE
  even when they would not normally do so when addressed
  to the non-data-server personality.  Stateids must obey
  the rules of the data server in that any use of special
  stateids or stateids with non-zero seqid values must 
  result in NFS4ERR_BAD_STATEID.
 </t>	
 <t>
  Until the server first executes an operation from class 2
  or class 3, the client MUST not depend on the operation 
  being executed by either the data-server or the non-data-server
  personality.  The server MUST pick one personality consistently
  for a given COMPOUND, with the only possible transition being 
  a single one when the first operation from class 2 or class 3
  is executed.
 </t>
 <t>
  Because of the complexity induced by assigning filehandles so
  they can be used on both a data server and a metadata server, it
  is recommended that where the same server can have both 
  personalities, the server assign filehandles within layouts so 
  that separate filehandles are applied to both personalities.
  This makes unambiguous the server for which a given request
  is intended.
 </t>

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Monday, November 05, 2007 4:54 PM
To: Noveck, Dave
Cc: Muntz, Daniel; [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some

Robert..

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 08 Nov 2007 11:48:22 -0600

> There shouldn't be any problem with adding that, right?

So it appears from the objects I-D that the information
being transferred to the server on LAYOUTCOMMIT is the error
list.  Is there a reason that this needs to move from LAYOUTCOMMIT
to LAYOUTRETURN or why the client would use LAYOUTCOMMIT in the
case of WRITEs vs. LAYOUTRETURN for READs?  Obviously, I am
in need of clarity about how layout type specific data will be
used on LAYOUTRETURN, etc.

> I can't be specific about the requirements but they don't require a

You can't be specific because you don't know the requirements or
are unable to share them?

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Thu, 08 Nov 2007 21:10:20 +0200

> used on LAYOUTRETURN, etc.

lou_body is currently defined as follows in pnfs-obj-04:

   struct pnfs_osd_layoutupdate4 {
       pnfs_osd_deltaspaceused4    delta_space_used;
       pnfs_osd_ioerr4             ioerr<>;
   };

My plan is to keep only delta_space_used in pnfs_osd_layoutupdate4
for usage at LAYOUTCOMMIT time and move the ioerr to the LAYOUTRETURN
args which may be used both for reads or writes.

The idea is that if the client hit an I/O error it is going to return
the layout anyway and will report the error details in the process.

Benny

-- 
Benny Halevy
Software Architect
Tel/Fax: +972-3-647-8340
Mobile: +972-54-802-8340
US:      +1-412-203-3187
[email-masked]
 
Panasas, Inc.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 08 Nov 2007 13:46:18 -0600

> the layout anyway and will report the error details in the process.

Okay, thanks for the clarification.  It seems reasonable if
we are moving the errors to the LAYOUTRETURN and thus the
layout is being returned in tandem.  Better place for it.

I will work up a set of diffs and post them before inclusion in
the I-D.  This will include the removal of the loca_time_access
field from LAYOUTCOMMIT.

Spencer

---

From: Robert Gordon <[email-masked]>
Date: Thu, 08 Nov 2007 14:42:52 -0600

On Nov 8, 2007, at 10:58 AM, Noveck, Dave wrote:

>  Clients MUST use the filehandle contained

does this mean there is a corresponding change to remove
this :-

[snip]

	of elements in nfl_fh_list depends on whether sparse or dense packing  
is

If not, then my preference would be to leave the sentence as it is now:

" Clients MUST use the filehandle described within the layout when
   accessing data on NFSv4.1 data servers."

Robert.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 08 Nov 2007 15:40:19 -0600

> the I-D.  This will include the removal of the loca_time_access

Okay, here is what I have.  Remove loca_time_access as noted
by the "diff" notation below:

  struct LAYOUTCOMMIT4args {
          /* CURRENT_FH: file */
          offset4                 loca_offset;
          length4                 loca_length;
          bool                    loca_reclaim;
          stateid4                loca_stateid;
          newoffset4              loca_last_write_offset;
          newtime4                loca_time_modify;
-        newtime4                loca_time_access;
          layoutupdate4           loca_layoutupdate;
};

For LAYOUTRETURN, I have added the layouttype4 specific
data to the "layoutreturn4" struct.

struct LAYOUTRETURN4args {
         /* CURRENT_FH: file */
         bool                    lora_reclaim;
         layoutreturn_stateid    lora_recallstateid;
         layouttype4             lora_layout_type;
         layoutiomode4           lora_iomode;
         layoutreturn4           lora_layoutreturn;
};

This is only done in the "file" case.  The FSID and ALL
cases of layoutreturn4 will not have layout specific data.

struct layoutreturn_file4 {
          offset4         lrf_offset;
          length4         lrf_length;
          stateid4        lrf_stateid;
+%       /* layouttype4 specific data */
+        opaque          lrf_body<>;
};

union layoutreturn4 switch(layoutreturn_type4 lr_returntype) {
          case LAYOUTRETURN4_FILE:
                  layoutreturn_file4      lr_layout;
          default:
                  void;
};

Unless there are objections, this will show up in the next I-D.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 9 Nov 2007 10:26:49 -0500

How about "specified by the layout"? 

-----Original Message-----
From: Robert Gordon [mailto:[email-masked]] 
Sent: Thursday, November 08, 2007 3:43 PM
To: Noveck, Dave
Cc: Muntz, Daniel; [email-masked]
Subject: Re: [nfsv4] Proposed resolutions for some

On Nov 8, 2007, at 10:58 AM, Noveck, Dave wrote:

>  Clients MUST use the filehandle contained  within the layout when 

does this mean there is a corresponding change to remove this :-

[snip]

Robert.

---

From: Robert Gordon <[email-masked]>
Date: Fri, 09 Nov 2007 11:42:52 -0600

> How about "specified by the layout"?

Love it ! :-)

Robert.