Subject: RE: [nfsv4] Layout stateids

From: [email-masked]
Date: Mon, 6 Aug 2007 10:43:52 -0400

I've been evaluating how the block layout driver for pNFS would make use
of stateids in the layout operations.  (i.e LAYOUTGET, LAYOUTRETURN,
LAYOUTCOMMIT and CB_LAYOUTRECALL).  My conclusion is that the stateid is
superfluous and that it does not enhance the protocol in any significant
way.  

>From the proposal, I can find two justifications for using a stateid in
the layout.  Neither one is convincing.  The two reasons are:
	1. Force via the client to issue an OPEN before issuing a
LAYOUTGET 	   operation. 
	2. To sequence the operations between client and metadata
server. 

What purpose does it serve to force an open before issuing a LAYOUTGET?
The inclusion of state id as an argument of LAYOUTGET enforces this
requirement for the first LAYOUTGET for a given file, but given that
LAYOUTs are valid even after a CLOSE operation, subsequent LAYOUTCOMMIT,
LAYOUTRETURN and LAYOUTGET operations can use the stateid returned by
the last LAYOUTGET, LAYOUTCOMMIT, LAYOUTRETURN or CB_LAYOUTRECALL.  Are
these operations meant to be rejected with INVALID_STATEID?  If so, then
it seems odd that the spec explicitly allows for the layout to be held
after a CLOSE.  If not, I cannot see what the purpose of using the
stateid on the first LAYOUTGET, since subsequent LAYOUTGET operations
cannot be validated via the stateid.

As far as using the stateid to sequence operations between the client
and server, it unnecessary, and in the most complex situations does not
help.  For example, during the 69th IETF meeting we discussed the race
associated with a compound LOOKUP, PUTFH, OPEN, LAYOUTGET request and a
CB_LAYOUTRECALL.  Sessions provide the framework to handling this, and
stateids do not.

I'm suggesting that we need further justification before including
stateids in layouts.

-Jason

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Friday, July 27, 2007 1:09 AM
To: [email-masked]
Subject: [nfsv4] Layout stateids

Working from Mike's stateids in layouts proposal, the ensuing thread
of discussion and then the continued discussion during Monday's WG
meeting, I offer the following proposal for stateids in layouts.

The main purpose of this change is to ensure the client has a layout
that is correctly derived from a valid OPEN of the file and to tie the
layout to the lease held by the client.  Of secondary concern is to
increase the potential for parallelism between LAYOUTGET and
LAYOUTRETURN, and to ease the data server's validation of layouts.

Some assumptions/observations:

Layouts are similar to byte range file locks; they are range based,
can merge, can be split.  Thus, the ordering of LAYOUTGET and
LAYOUTRETURN matters greatly to the shared view the client and server
has for layout ranges.  The addition of stateids for LAYOUTGET and
LAYOUTRETURN will allow for the ordering of execution at the server to
be communicated to the client via the stateid4.seqid field just like
the OPEN and LOCK sequencing is communciated.  Note that the iomode
provide two distinct sets of layout ranges -- an upgrade path from the
READ iomode to the READ/WRITE iomode does NOT exist.

Layouts are similar to delegations in that the stateid is valid beyond
the life of the OPEN stateid (file can be closed and the client can
continue holding a valid layout stateid).  The other similarity with
delegations is that layout stateids are associated with the
filehandle/client pairing (NOT filehandle/"layout owner" like byte
range file locks).
	
The presence of layout stateids does not remove the utility of
Sessions' slot referencing mechanism on CB_SEQUENCE.  This was the
conclusion of the discussion during the NFSv4 WG meeting on Monday
(Brent Welch may send an email with a discussion of the general issues
of sequencing and race conditions).

Proposal:

LAYOUTGET will take a stateid and return one in the results.

For the first LAYOUTGET, the client MUST use a valid stateid as
returned by OPEN, OPEN_DOWNGRADE, LOCK, LOCKU, WANT_DELEGATION (open,
lock, or delegation stateids).  For subsequent LAYOUTGETs, the client
will provide a layout stateid.

The client MUST hold a valid layout for I/O to any storage
device type.

The use of special stateids for data server I/O is not allowed.

Layout stateids follow the sequencing and use rules of all other
stateid types.

Same stateid4.other value is returned for all layouts on the file
(e.g. multiple LAYOUTGETs and LAYOUTRETURNs to the same file).

LAYOUTRETURN will take a layout stateid and return a stateid.  If the
server has recalled a CB_LAYOUTRECALL and the client is returning a
layout as a result, the client will also provide the layout stateid
specified in the CB_LAYOUTRECALL.

LAYOUTCOMMIT will take a layout stateid; it will not return a stateid.

Layout stateids persist across CLOSE and DELEGRETURN.

As noted above, for the NFSv4.1 layout type, READ and WRITE to a data
server will continue to require an open, lock, or delegation stateid.
The data server SHOULD be able to map these stateids to the granted
layout to make sure the client has a layout, but in any case the data
server MUST reject I/Os to files that do not have layouts.

As with other state, layout stateids might become invalid
when the lease expires.

Spencer

-------
struct LAYOUTCOMMIT4args {
	/* CURRENT_FH: file */
	offset4			loca_offset;
	length4			loca_length;
	bool			loca_reclaim;
+	stateid4		loca_stateid;
	newoffset4		loca_last_write_offset;
	newtime4		loca_time_modify;
	newtime4		loca_time_access;
	layoutupdate4		loca_layoutupdate;
};
...
struct LAYOUTGET4args {
	/* CURRENT_FH: file */
	bool			loga_signal_layout_avail;
	layouttype4		loga_layout_type;
	layoutiomode4		loga_iomode;
	offset4			loga_offset;
	length4			loga_length;
	length4			loga_minlength;
+	stateid4		loga_stateid;
	count4			loga_maxcount;
};
struct LAYOUTGET4resok {
          bool               logr_return_on_close;
+	stateid4	   logr_stateid;
          layout4		   logr_layout;
};
union LAYOUTGET4res switch (nfsstat4 logr_status) {
case NFS4_OK:
	LAYOUTGET4resok	    logr_resok4;
case NFS4ERR_LAYOUTTRYLATER:
	bool		    logr_will_signal_layout_avail;
default:
	void;
};
+union layoutrecall_stateid switch (bool lrs_present) {
+case TRUE:
+	stateid4		lrs_stateid;
+case FALSE:
+	void;
+};
+
struct LAYOUTRETURN4args {
	/* CURRENT_FH: file */
	bool			lora_reclaim;
+	stateid4		lora_stateid;
+	recall_stateid		lora_recallstateid;
	layouttype4		lora_layout_type;
	layoutiomode4		lora_iomode;
	layoutreturn4		lora_layoutreturn;
};
struct LAYOUTRETURN4res {
	nfsstat4	    lorr_status;
+	stateid4	    lorr_stateid;
};
...
enum layoutrecall_type4 {
	LAYOUTRECALL4_FILE = 1,
	LAYOUTRECALL4_FSID = 2,
	LAYOUTRECALL4_ALL  = 3
};
struct layoutrecall_file4 {
	nfs_fh4		lor_fh;
	offset4		lor_offset;
	length4		lor_length;
+	stateid4	lor_stateid;
};
union layoutrecall4 switch(layoutrecall_type4 recalltype) {
case LAYOUTRECALL4_FILE:
	layoutrecall_file4 lor_layout;
case LAYOUTRECALL4_FSID:
	fsid4		   lor_fsid;
case LAYOUTRECALL4_ALL:
	void;
};
struct CB_LAYOUTRECALL4args {
	layouttype4		clora_type;
	layoutiomode4		clora_iomode;
	bool			clora_changed;
	layoutrecall4		clora_recall;
};

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4