Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

From: Jason Glasgow <[email-masked]>
Date: Thu, 2 Apr 2009 22:33:47 -0400

In accordance with approach 1 outlined by David Black after the 74th IETF
meeting, I propose the following change to the
draft-ietf-nfsv4-pnfs-block-12.txt spec:

OLD:
   The client MUST use SETATTR with a layout hint of type
   LAYOUT4_BLOCK_VOLUME to inform the server of its maximum I/O time
   prior to issuing the first LAYOUTGET operation.  The maximum io time
   hint is a per client attribute, and as such the server SHOULD
   maintain the value set by each client.

NEW:
   The client MUST use SETATTR with a layout hint of type
   LAYOUT4_BLOCK_VOLUME to inform the server of its maximum I/O time
   prior to issuing the first LAYOUTGET operation.  The NFS spec
   [NFSV4.1] defines the layout hint as a per file attribute.
   In the case of the block client, the server MUST maintain
   this value separately for each client.  If the client does not
   specify this attribute on a file for which a block layout is
   requested, the server SHOULD use the most recent value provided by
   the same client for any file.  Unless the maximum I/O time changes,
   the client SHOULD send the layout hint only once to the server.

Comments?

Thanks,
-Jason

On Wed, Mar 25, 2009 at 4:37 PM, <[email-masked]> wrote:

> This message summarizes the problem and the results of discussion
> in the nfsv4 WG meeting this morning in San Francisco. The slides
> should appear in the online meeting materials soon - both the
> Problem and Possible Approaches material below is from the slides.
>
> Problem:
> - layout_hint attribute is pNFS-layout-specific
> - File and Object layouts:
>  - Tell server how to create file (e.g., striping)
> - Block layout:
>  - Tell server what client's max I/O time is
> - Problem discovered by implementers:
>  - Client max I/O time is per-client
>  - layout_hint is specified as per-file
>  - Setting this on every file is overkill for block layout
>
> The first outcome of the discussion was that something needs
> to be done, because what the pNFS block layout draft currently
> says is not consistent with the main NFSv4.1 draft.
>
> Possible Approaches:
> 1. Tweak default behavior if attribute not set
>   - Use most recent value provided by same client
> 2. Make attribute scope layout-dependent
>   - File and object layouts: per-file
>   - Block layout: per-filesystem (track per client)
> 3. Define a new attribute
>
> The sense of the discussion was that the above list appears
> to be in order of preference, as a new attribute is clearly
> NFSv4.2 territory, not 4.1, and it's preferable to keep the
> changes limited to the block layout draft.  That makes
> approach 1 preferable, or even just enough of a change to
> align the block draft with the main NFSv4.1 draft.  One
> consequence of this discussion is that the attribute will
> remain per-file in the main NFSv4.1 draft.
>
> Plan:
> (A) The pNFS block draft authors will write up text changes
>        for both the minimum required correction and the default
>        behavior (i.e., what the server is supposed to do if a
>        client doesn't specify this attribute on a file for which
>        a block layout is requested).  That text will be sent to
>        this list by a week from this Friday (i.e., by Friday,
>        April 3).
> (B) The NFSv4 WG chairs plan to conduct a 1-week review of this
>        text.  The final text changes will be sent to the RFC
>        Editor for incorporation into the pNFS block draft before
>        it's published as an RFC.
>
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Distinguished Engineer
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 3 Apr 2009 06:29:10 -0400

"the first LAYOUTGET" leaves some things unspecified.  Specifically, if
the server reboots does a new SETATTR need to be done?  What if the
client reboots?
 
> Unless the maximum I/O time changes,
 
I'm assuming here that "SHOULD .... only once" is equivalent to "SHOULD
NOT ... more than once".
 
Do you need something this strong?  Wouldn't it be enough to say the
client need not send it multiple times?  For example, if you say I MUST
send it before the first LAYOUTGET and SHOULD NOT send it twice then I
have to serialize.  If I don't have the SHOULD NOT, then I can add a
SETATTR to all the compounds that do a LAYOUTGET before I get the first
reply.  That may mean the server gets multiple SETATTR's but I think
this is a no-harm-no-foul situation.
 
I believe "should ... only once" would be fine.

________________________________

From: Jason Glasgow [mailto:[email-masked]] 
Sent: Thursday, April 02, 2009 10:34 PM
To: [email-masked]; [email-masked]
Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

In accordance with approach 1 outlined by David Black after the 74th
IETF meeting, I propose the following change to the
draft-ietf-nfsv4-pnfs-block-12.txt spec:

OLD:

NEW:

Comments?

Thanks,
-Jason

	This message summarizes the problem and the results of
discussion
	in the nfsv4 WG meeting this morning in San Francisco. The
slides
	should appear in the online meeting materials soon - both the
	Problem and Possible Approaches material below is from the
slides.
	
	Problem:

	Possible Approaches:

	Plan:
	
	Thanks,
	--David

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 3 Apr 2009 07:28:42 -0700 (PDT)

If the client ignores the SHOULD (perhaps for the reasons Dave Noveck
gave ... the client doesn't want to maintain a table with three columns:

server ID
last time a hint was sent
last time the the server restarted

I know if I was implementing a client, I wouldn't want to maintain this
table), what bad things will happen?

Also, there is this text in the i_D, which David Black said needs to be
fixed:

   The maximum io time

I suggest,

   While the maximum io time
   hint is a per file attribute, it is actually a per-client,
   characteristic. Thus the server MUST note the last
   maximum io time hint sent from the client. Each time the
   maximum io time changes, the server must apply it to all files that
   the client has a layout for.

On Thu, April 2, 2009 7:33 pm, Jason Glasgow wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

-- 
Mike Eisler, Senior Technical Director, NetApp, 719 599 9026,
http://blogs.netapp.com/eislers_nfs_blog/

---

From: Jason Glasgow <[email-masked]>
Date: Fri, 3 Apr 2009 10:47:51 -0400

On Fri, Apr 3, 2009 at 6:29 AM, Noveck, Dave <[email-masked]> wrote:

> the server reboots does a new SETATTR need to be done?  What if the client
> reboots?
>
The answer is that whenever a new session is established.  If the server
reboots?  Depends on whether or not the server implements persistent
sessions.  Without persistent sessions certainly yes a new SETATTR needs to
be sent.  If the client reboots?  I think the easiest client implementation
is to send the layout hint again.  (See change below).

> I'm assuming here that "SHOULD .... only once" is equivalent to "SHOULD NOT
> ... more than once".
>
> Do you need something this strong?  Wouldn't it be enough to say the client
> need not send it multiple times?  For example, if you say I MUST send it
> before the first LAYOUTGET and SHOULD NOT send it twice then I have to
> serialize.  If I don't have the SHOULD NOT, then I can add a SETATTR to all
> the compounds that do a LAYOUTGET before I get the first reply.  That may
> mean the server gets multiple SETATTR's but I think this is a
> no-harm-no-foul situation.
>
I agree that the client should not have to serialize.  I am willing to
change to 'should', but I think  'SHOULD NOT send a layout hint with every
LAYOUTGET' is much closer to the intended meaning.

The new proposal is thus:

OLD:

NEW:
   After a new session is established,
   the client MUST use SETATTR with a layout hint of type
   LAYOUT4_BLOCK_VOLUME to inform the server of its maximum I/O time
   prior to issuing the first LAYOUTGET operation.  The NFS spec
   [NFSV4.1] defines the layout hint as a per file attribute.
   In the case of the block client, the server MUST maintain
   this value separately for each client.  If the client does not
   specify this attribute on a file for which a block layout is
   requested, the server SHOULD use the most recent value provided by
   the same client for any file.  The client SHOULD NOT send a SETATTR
   of the layout hint with every LAYOUTGET.

>
>  ------------------------------
> *From:* Jason Glasgow [mailto:[email-masked]]
> *Sent:* Thursday, April 02, 2009 10:34 PM
> *To:* [email-masked]; [email-masked]
> *Subject:* Re: [nfsv4] pNFS block layout & layout_hint attribute
>

-- 
Jason

-- 
Jason

---

From: Jason Glasgow <[email-masked]>
Date: Fri, 3 Apr 2009 11:03:19 -0400

> table), what bad things will happen?

The intention is prevent client from sending a SETATTR with a layout hint
prior to every LAYOUTGET operation.  Will bad things happen?  No.  It will
just waste bandwidth and cause the server additional work processing an
unneeded operation.

>   the client has a layout for.

That phrase "per client attribute" was removed from the OLD and replace with
the NEW (see below).  Combining your suggestion with the previous changes
based on Dave's comments yields:

OLD:

NEW:
   After a new session is established,
   the client MUST use SETATTR with a layout hint of type
   LAYOUT4_BLOCK_VOLUME to inform the server of its maximum I/O time
   prior to issuing the first LAYOUTGET operation.
   While the maximum io time
   hint is a per file attribute, it is actually a per-client,
   characteristic. Thus the server MUST maintain the last
   maximum io time hint sent separately for each client. Each time the
   maximum io time changes, the server must apply it to all files that
   the client has a layout for.  If the client does not
   specify this attribute on a file for which a block layout is
   requested, the server SHOULD use the most recent value provided by
   the same client for any file.  The client SHOULD NOT send a SETATTR
   of the layout hint with every LAYOUTGET.

---

From: [email-masked]
Date: Wed, 8 Apr 2009 22:46:34 -0400

Double-check - this is the text modification that the
block layout draft authors intend to go with.  Speak now
if you want to see changes (this change is in Section
2.3.8 of the block layout draft).  I believe one of the
WG chairs needs to determine the WG "rough consensus"
for this text ... (hint) ...
 
OLD:

NEW:

Thanks,
--David

	From: [email-masked] [mailto:[email-masked]] On
Behalf Of Jason Glasgow
	Sent: Friday, April 03, 2009 11:03 AM
	To: Mike Eisler
	Cc: Black, David; [email-masked]
	Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

	On Fri, Apr 3, 2009 at 10:28 AM, Mike Eisler
<[email-masked]> wrote:

		If the client ignores the SHOULD (perhaps for the
reasons Dave Noveck
		gave ... the client doesn't want to maintain a table
with three columns:
		
		server ID
		
		I know if I was implementing a client, I wouldn't want
to maintain this
		table), what bad things will happen?

	The intention is prevent client from sending a SETATTR with a
layout hint prior to every LAYOUTGET operation.  Will bad things happen?
No.  It will just waste bandwidth and cause the server additional work
processing an unneeded operation.

		Also, there is this text in the i_D, which David Black
said needs to be
		fixed:

		  The maximum io time
		  hint is a per client attribute, and as such the server
SHOULD
		  maintain the value set by each client.

		I suggest,
		
		  While the maximum io time
		  hint is a per file attribute, it is actually a
per-client,
		  characteristic. Thus the server MUST note the last
		  maximum io time hint sent from the client. Each time
the
		  maximum io time changes, the server must apply it to
all files that
		  the client has a layout for.

	That phrase "per client attribute" was removed from the OLD and
replace with the NEW (see below).  Combining your suggestion with the
previous changes based on Dave's comments yields:

	OLD:
	   The client MUST use SETATTR with a layout hint of type
	   LAYOUT4_BLOCK_VOLUME to inform the server of its maximum I/O
time
	   prior to issuing the first LAYOUTGET operation.  The maximum
io time
	   hint is a per client attribute, and as such the server SHOULD
	   maintain the value set by each client.
	
	NEW:
	
	   the client MUST use SETATTR with a layout hint of type
	   LAYOUT4_BLOCK_VOLUME to inform the server of its maximum I/O
time
	   prior to issuing the first LAYOUTGET operation. 
	   While the maximum io time
	   hint is a per file attribute, it is actually a per-client,
	   characteristic. Thus the server MUST maintain the last
	   maximum io time hint sent separately for each client. Each
time the
	   maximum io time changes, the server must apply it to all
files that
	   the client has a layout for.  If the client does not

	   requested, the server SHOULD use the most recent value
provided by
	
	   the same client for any file.  The client SHOULD NOT send a
SETATTR
	   of the layout hint with every LAYOUTGET.

The maximum io time
		>    prior to issuing the first LAYOUTGET operation.
The NFS spec

---

From: Benny Halevy <[email-masked]>
Date: Thu, 09 Apr 2009 12:14:00 +0300

>    prior to issuing the first LAYOUTGET operation.

What should/must the server do if the client fails to do so?
My intuition is that some clients wouldn't be able to know
their exact max I/O time.  Should they make it up or can the
server be configured in such a way that would allow it to
still work with such clients.

Does that MUST hold true if the client has multiple sessions
or would it be sufficient to set the layout hint only for the
first sessions it establishes with the server?

Benny

>     ------------------------------------------------------------------------
>     *From:* [email-masked] [mailto:[email-masked]] *On
>     Behalf Of *Jason Glasgow
>     *Sent:* Friday, April 03, 2009 11:03 AM
>     *To:* Mike Eisler
>     *Cc:* Black, David; [email-masked]
>     On Fri, Apr 3, 2009 at 10:28 AM, Mike Eisler <[email-masked]
>     <mailto:[email-masked]>> wrote:
> 
>     The intention is prevent client from sending a SETATTR with a layout
>     hint prior to every LAYOUTGET operation.  Will bad things happen? 
>     No.  It will just waste bandwidth and cause the server additional
>     work processing an unneeded operation.
> 
>      
> 
>     replace with the NEW (see below).  Combining your suggestion with
>     the previous changes based on Dave's comments yields:
> 
> 
>     OLD:
>         > On Wed, Mar 25, 2009 at 4:37 PM, <[email-masked]
>         <mailto:[email-masked]>> wrote:
>         >
> ------------------------------------------------------------------------
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 9 Apr 2009 06:31:14 -0400

> What should/must the server do if the client fails to do so?

Good question. I assumed it was that it would not grant layouts.  Is
that what everybody else was thinking?  Should this (or soemthing else)
be made explicit?

> their exact max I/O time.  

I didn't think you needed your exact max io time, but just some reliable
not-to-be exceeded value.

> Should they make it up or can the server be configured in such 
> a way that would allow it to still work with such clients.

IIRC, the max io time had data integrity implications.  True?  You can't
stop people from sending bullshit values but the protocol spec shouldn't
do anything that would encourage it.  The current text has a "MUST" and
configuring the server to accept clients who violate the protocol
doesn't sound like a good thing.

> first sessions it establishes with the server?

Clearly, the text current text says the client must send it.

Are you suggesting a change from that?

If so, why?

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, April 09, 2009 5:14 AM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

On Apr. 09, 2009, 5:46 +0300, [email-masked] wrote:
> Double-check - this is the text modification that the block layout 
> draft authors intend to go with.  Speak now if you want to see changes

> (this change is in Section
> 2.3.8 of the block layout draft).  I believe one of the WG chairs 
> needs to determine the WG "rough consensus"

What should/must the server do if the client fails to do so?
My intuition is that some clients wouldn't be able to know their exact
max I/O time.  Should they make it up or can the server be configured in
such a way that would allow it to still work with such clients.

Does that MUST hold true if the client has multiple sessions or would it
be sufficient to set the layout hint only for the first sessions it
establishes with the server?

Benny

>         gave ... the client doesn't want to maintain a table with
three
>         columns:
layout
SHOULD
>           maximum io time changes, the server must apply it to all
files
>         that
time
time
SETATTR
SHOULD
>         >    the same client for any file.  Unless the maximum I/O
time
>         changes,
discussion
slides

---

From: Benny Halevy <[email-masked]>
Date: Thu, 09 Apr 2009 17:30:33 +0300

> that what everybody else was thinking?  Should this (or soemthing else)

Taking an example from section 2.3.8. Client Fencing, it makes sense
that the server SHOULD return NFS4ERR_LAYOUTUNAVAILABLE to LAYOUTGET
if the client never set its maximum I/O time as if it set it to
the unbounded (all 1's) value.  I.e. these two cases can be defined
as equivalent, making the unbounded time the default value.

> doesn't sound like a good thing.

Agreed.  My question is whether that MUST is realistic.
I guess that configuring the server with a particular number is similar
to configure each client to set that same number, though it seems
like an easier task from the administrator point of view.

That said, defining the default value as "unbounded", changing the client
MUST to SHOULD and allowing servers that do not support fencing to work
under these conditions seems to me like a practical approach.

> Are you suggesting a change from that?

I see no value in setting it per session since the value is per-client,
not per session.

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 9 Apr 2009 14:38:13 -0400

> Agreed.  My question is whether that MUST is realistic.

Got me.  If you feel this is a problem, then you and David
can argue about it.

> > Are you suggesting a change from that?

I guess you are saying that the answer is "yes".

> I see no value in setting it per session since the value 
> is per-client, not per session.

This would require sending one more op (and not more compound) in cases
in which there are multiple sessions for a single.  It's in the noise.
I assume the author found this simplification helpful.  Maybe you would
have done it differently.  But given where we are in this process, I
don't think the question should be "would you have done it
differently?".   I think the question has to be "Is there some reason
that you think what is specified is a significant problem?"

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, April 09, 2009 10:31 AM
To: Noveck, Dave
Cc: [email-masked]; [email-masked]
Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

That said, defining the default value as "unbounded", changing the
client
MUST to SHOULD and allowing servers that do not support fencing to work
under these conditions seems to me like a practical approach.

Benny

SETATTR
>>         If the client ignores the SHOULD (perhaps for the reasons
Dave
>>         Noveck

>>        prior to issuing the first LAYOUTGET operation.  The maximum
io
time
>>        requested, the server SHOULD use the most recent value
provided
> by
>>         >    specify this attribute on a file for which a block
layout
> is
>>         >>        behavior (i.e., what the server is supposed to do
if
> a
----------------------------------------------------------------------

---

From: Benny Halevy <[email-masked]>
Date: Fri, 10 Apr 2009 10:06:39 +0300

> I guess you are saying that the answer is "yes".

Yes.  I do suggest changing the requirement for the client to
set the layout hint from MUST to SHOULD and defining the server's
behavior in case the client did not set the layout hint to be the
same as currently defined for the case where the client set the
maximum io time via the layout hint to the "unbound" value.

>> is per-client, not per session.

(note, this answer is for a different question, regarding setting
the layout hint once vs. for each new session)

> that you think what is specified is a significant problem?"

Yeah, I agree that setting the layout hint per session is not a big deal.

Benny

---

From: "Mike Eisler" <[email-masked]>
Date: Fri, 10 Apr 2009 06:54:47 -0700 (PDT)

> Yeah, I agree that setting the layout hint per session is not a big deal.

There's an implication with the text as proposed that the server
is going to maintain this per session ID and not per client ID.

If I/O time is a session characteristic, then state it as such.
If I/O time is a client characterisitic, then:

s/a new session/a new client ID/

---

From: "Welch, Brent" <[email-masked]>
Date: Fri, 10 Apr 2009 08:04:51 -0700

I agree with Benny.  I would rather see this as SHOULD with a description
of what the server does when the client doesn't set the attribute.
This is all about tuning the grace period on the server, and it seems
easier to set a default on the server than to require every block client.
I'd rather see the protocol work without it, and then allow clients to make
the call in order to optimize the behavior during the grace period.

>>>Benny Halevy said:

--
Brent Welch
Director, Software Architecture, Panasas Inc
The Leader in Parallel Storage

www.panasas.com
[email-masked]

---

From: Jason Glasgow <[email-masked]>
Date: Fri, 10 Apr 2009 11:31:43 -0400

I will make the change as suggested

> s/a new session/a new client ID/.

Despite Benny and Brent's arguments, with respect to changing from MUST to
SHOULD, I believe the text should stay as it is currently written..  By
forcing a client with an unbounded maximum I/O time to send a LAYOUT hint,
the client can clearly establish whether or not the server will accept
LAYOUTGET operations.  If we change the langauge from MUST to SHOULD, we
open the door to clients repeatedly sending LAYOUTGET operations to a server
which does not implement LUN masking, and repeatedly getting error
responses.  This is suboptimal behavior.  (It should be addressed by adding
another error code to the NFSv41 specification, but I would rather not go
there.)

Making the change suggested by Mike, the proposed text change is:

OLD:

NEW:
   After a new client ID is established,

-Jason

---

From: [email-masked]
Date: Fri, 10 Apr 2009 14:25:21 -0400

I agree with Jason that a "MUST" on the client to send a value
is the right approach.  Anything weaker risks requiring client
knowledge of the underlying infrastructure that is unreasonable
to assume that client implementers will have in general (e.g.,
the sheer variety of infrastructure on which the Linux client
may run boggles the mind).
 
It looks like we need text telling the server what to do with a
client that gets this wrong, so I'd expand:
 
   If the client does not
   specify this attribute on a file for which a block layout is
   requested, the server SHOULD use the most recent value
   provided by the same client for any file. 
 
to:
 
   If the client does not
   specify this attribute on a file for which a block layout is
   requested, the server SHOULD use the most recent value
   provided by the same client for any file; if that client has
   not provided a value for this attribute, the server SHOULD
   reject the layout request with the error NFS4ERR_LAYOUTUNAVAILABLE.
 
The rationale for the "SHOULD" rather than a "MUST" for this server
behavior is that a stronger case can be made for the server
understanding (and in particular being administratively configured
to correctly understand) the underlying infrastructure, making an
RFC 2119 "SHOULD" warning appropriate (i.e., if the server isn't
sure about what's going on, the right [and safe] thing to do is
reject the layout request from the clue-impaired client).

Thanks,
--David

	From: [email-masked] [mailto:[email-masked]] On
Behalf Of Jason Glasgow
	Sent: Friday, April 10, 2009 11:32 AM
	To: Mike Eisler
	Cc: [email-masked]
	Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

	Despite Benny and Brent's arguments, with respect to changing
from MUST to SHOULD, I believe the text should stay as it is currently
written..  By forcing a client with an unbounded maximum I/O time to
send a LAYOUT hint, the client can clearly establish whether or not the
server will accept LAYOUTGET operations.  If we change the langauge from
MUST to SHOULD, we open the door to clients repeatedly sending LAYOUTGET
operations to a server which does not implement LUN masking, and
repeatedly getting error responses.  This is suboptimal behavior.  (It
should be addressed by adding another error code to the NFSv41
specification, but I would rather not go there.)
	
	Making the change suggested by Mike, the proposed text change
is:

	OLD:
	
	NEW:

	

	-Jason

	On Fri, Apr 10, 2009 at 9:54 AM, Mike Eisler
<[email-masked]> wrote:

		> Yeah, I agree that setting the layout hint per session
is not a big deal.

		There's an implication with the text as proposed that
the server
		is going to maintain this per session ID and not per
client ID.
		
		If I/O time is a session characteristic, then state it
as such.
		If I/O time is a client characterisitic, then:

per-client,
		>>>>    maximum io time hint sent separately for each
client. Each time
		>> the
		>>>>    maximum io time changes, the server must apply
it to all files

---

From: Benny Halevy <[email-masked]>
Date: Fri, 10 Apr 2009 23:04:48 +0300

> reject the layout request from the clue-impaired client).

Yup, that seems reasonable.

Benny

>     *Sent:* Friday, April 10, 2009 11:32 AM
>     *To:* Mike Eisler
>     *Cc:* [email-masked]
>     Despite Benny and Brent's arguments, with respect to changing from
>     MUST to SHOULD, I believe the text should stay as it is currently
>     send a LAYOUT hint, the client can clearly establish whether or not
>     the server will accept LAYOUTGET operations.  If we change the
>     langauge from MUST to SHOULD, we open the door to clients repeatedly
>     sending LAYOUTGET operations to a server which does not implement
>     LUN masking, and repeatedly getting error responses.  This is
>     suboptimal behavior.  (It should be addressed by adding another
>     error code to the NFSv41 specification, but I would rather not go
>     there.)
> 
>     On Fri, Apr 10, 2009 at 9:54 AM, Mike Eisler <[email-masked]
>         >>>>    maximum io time hint sent separately for each client.
>         Each time
>         >> the
>         [email-masked] <mailto:[email-masked]>

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 22 Apr 2009 00:37:30 -0500

>    of the layout hint with every LAYOUTGET.

(with WG co-chair hat on)

In re-reading this thread I am not sure I can effectively discern the
final, proposed text and bluntly don't know if we have a rough
consensus.

With this note that all of the related NFSv4.1 I-Ds are ready and
waiting in the editor's queue.

Let's close on this quickly and move them forward.   If we can't reach
rough consensus, I will ask that the I-Ds be moved forward as is and
we start the bis process for the blocks work to drive the details
to closure in an environment that is not hurried.

Spencer

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Apr 2009 08:37:01 -0400

> Let's close on this quickly and move them forward.

I agree.

> If we can't reach rough consensus, I will ask that 
> the I-Ds be moved forward as is and we start the bis 
> process for the blocks work to drive the details to 
> closure in an environment that is not hurried.

My only issue is making sure that the attempt to reach rough consensus
and proceed with the alternate plan has a firm time limit.  My proposal
is the end of the week, defined as Midnight this Friday in Jason's time
zone.

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Wednesday, April 22, 2009 1:38 AM
To: Jason Glasgow
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

On Apr 10, 2009, at 10:31 AM, Jason Glasgow wrote:

> sending LAYOUTGET operations to a server which does not implement LUN 
> masking, and repeatedly getting error responses.  This is suboptimal 
> behavior.  (It should be addressed by adding another error code to the

> NFSv41 specification, but I would rather not go
> there.)
>

Let's close on this quickly and move them forward.   If we can't reach
rough consensus, I will ask that the I-Ds be moved forward as is and we
start the bis process for the blocks work to drive the details to
closure in an environment that is not hurried.

Spencer

---

From: Jason Glasgow <[email-masked]>
Date: Wed, 22 Apr 2009 11:04:33 -0400

Spencer,

Let me attempt to clear up  the concerns.  The final proposed text change is
included below.  I believe that this incorporates all the input from those
who have posted on the issue.

-Jason

OLD:

NEW:
   the same client for any file; if that client has
   The client SHOULD NOT send a SETATTR of the layout hint with
   every LAYOUTGET.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Apr 2009 17:16:32 -0400

The remaining issue is the one Brent and Benny raised.
 
Benny said (on 4/10 early AM):

	Yes. I do suggest changing the requirement for the client to set
the layout hint from MUST to SHOULD and defining the server's behavior
in case the client did not set the layout hint to be the same as
currently defined for the case where the client set the maximum io time
via the layout hint to the "unbound" value.

And Brent said (on 4/10 in the evening):

	I agree with Benny. I would rather see this as SHOULD with a
description of what the server does when the client doesn't set the
attribute. This is all about tuning the grace period on the server, and
it seems easier to set a default on the server than to require every
block client. I'd rather see the protocol work without it, and then
allow clients to make the call in order to optimize the behavior during
the grace period.

I think you, David Black and I have argued that the MUST is right.  3-2
is not rough consensus but I suppose 4-1 is.  If anyone moves from
SHOULD to MUST, we are set.  If people decide that SHOULD is right, I'm
not going to be a holdout.  

	-----Original Message-----
	From: Jason Glasgow [mailto:[email-masked]] 
	Sent: Wednesday, April 22, 2009 11:05 AM
	To: Spencer Shepler; [email-masked]
	Cc: Noveck, Dave
	Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

	Spencer,
	
	Let me attempt to clear up  the concerns.  The final proposed
text change is included below.  I believe that this incorporates all the
input from those who have posted on the issue.
	
	-Jason

	OLD:
	
	NEW:
	
	   the same client for any file; if that client has
	   not provided a value for this attribute, the server SHOULD
	   reject the layout request with the error
NFS4ERR_LAYOUTUNAVAILABLE.
	   The client SHOULD NOT send a SETATTR of the layout hint with
	   every LAYOUTGET.

		I agree.

		> closure in an environment that is not hurried.

		My only issue is making sure that the attempt to reach
rough consensus
		and proceed with the alternate plan has a firm time
limit.  My proposal
		is the end of the week, defined as Midnight this Friday
in Jason's time
		zone.

		-----Original Message-----
		From: Spencer Shepler [mailto:[email-masked]]
		Sent: Wednesday, April 22, 2009 1:38 AM
		To: Jason Glasgow
		Cc: [email-masked]
		Subject: Re: [nfsv4] pNFS block layout & layout_hint
attribute

		
		
		>    prior to issuing the first LAYOUTGET operation.
The maximum io
per-client,
		>    maximum io time hint sent separately for each
client. Each time the
		>    specify this attribute on a file for which a block
layout is

		In re-reading this thread I am not sure I can
effectively discern the
		final, proposed text and bluntly don't know if we have a
rough
		consensus.
		
		With this note that all of the related NFSv4.1 I-Ds are
ready and
		waiting in the editor's queue.
		
		Let's close on this quickly and move them forward.   If
we can't reach
		rough consensus, I will ask that the I-Ds be moved
forward as is and we
		start the bis process for the blocks work to drive the
details to
		closure in an environment that is not hurried.
		
		Spencer

---

From: Jason Glasgow <[email-masked]>
Date: Wed, 22 Apr 2009 17:33:33 -0400

I thought that Benny had said (
http://www.ietf.org/mail-archive/web/nfsv4/current/msg06945.html)

in reply to David's message on the subject.

-Jason

On Wed, Apr 22, 2009 at 5:16 PM, Noveck, Dave <[email-masked]>wrote:

>  Yes. I do suggest changing the requirement for the client to set the
> layout hint from MUST to SHOULD and defining the server's behavior in case
> the client did not set the layout hint to be the same as currently defined
> for the case where the client set the maximum io time via the layout hint
> to the "unbound" value.
>
> I agree with Benny. I would rather see this as SHOULD with a description of
> what the server does when the client doesn't set the attribute. This is all
> about tuning the grace period on the server, and it seems easier to set a
> default on the server than to require every block client. I'd rather see the
> protocol work without it, and then allow clients to make the call in order
> to optimize the behavior during the grace period.
>
> I think you, David Black and I have argued that the MUST is right.  3-2 is
> not rough consensus but I suppose 4-1 is.  If anyone moves from SHOULD to
> MUST, we are set.  If people decide that SHOULD is right, I'm not going to
> be a holdout.
>
>
>
> *From:* Jason Glasgow [mailto:[email-masked]]
> *Sent:* Wednesday, April 22, 2009 11:05 AM
> *To:* Spencer Shepler; [email-masked]
> *Cc:* Noveck, Dave
> Let me attempt to clear up  the concerns.  The final proposed text change
> is included below.  I believe that this incorporates all the input from
> those who have posted on the issue.
>
> -Jason
>
>
> OLD:

---

From: Spencer Shepler <[email-masked]>
Date: Wed, 22 Apr 2009 16:30:46 -0500

On Apr 22, 2009, at 4:16 PM, Noveck, Dave wrote:

> layout hint from MUST to SHOULD and defining the server's behavior  
> currently defined for the case where the client set the maximum io  
> time via the layout hint to the "unbound" value.
>
> attribute. This is all about tuning the grace period on the server,  
> and it seems easier to set a default on the server than to require  
> every block client. I'd rather see the protocol work without it, and  
> then allow clients to make the call in order to optimize the  
> behavior during the grace period.
>
> I think you, David Black and I have argued that the MUST is right.   
> 3-2 is not rough consensus but I suppose 4-1 is.  If anyone moves  
> from SHOULD to MUST, we are set.  If people decide that SHOULD is  
> right, I'm not going to be a holdout.
>
I don't have the reference right now but Benny did come
back and agree with David Black's last summary
and the justification for MUST.

Hence, this is why I have asked for what Jason provided (final summary)
and need everyone that expressed an opinion before to express
and opinion again.

Spencer

> Let me attempt to clear up  the concerns.  The final proposed text  
> change is included below.  I believe that this incorporates all the  
> LUN
> we

---

From: [email-masked]
Date: Wed, 22 Apr 2009 20:10:54 -0400

Spencer,

> and need everyone that expressed an opinion before to express

I'm still in favor of "MUST", and the block layout client
implementations that I know about are planning to send this
attribute.

OTOH, I'm more in favor of a conclusion that gets us to text that
deals with this situation, as both "MUST" and "SHOULD" for the new
approach are far superior to what's currently in the draft.  If it
takes "SHOULD" to get to a conclusion, I can live with that.

Thanks,
--David

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Spencer Shepler
> Sent: Wednesday, April 22, 2009 5:31 PM
> To: Noveck, Dave
> Cc: [email-masked]
> Each time the
> > > behavior.  (It should be addressed by adding another 
> error code to  
> > the
> >
> all files

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 22 Apr 2009 20:24:28 -0400

I read "Yup, that seems reasonable" as applying to the final paragaph of
David's message, rather than the entirety.  I guess the fact that the
singular was used led me to assume that only that one point was being
conceded.
 
I could be all wet, though.  I think the right answer is Benny telling
either what he meant or what his position on "SHOULD vs. MUST" is now..

From: Jason Glasgow [mailto:[email-masked]] 
Sent: Wednesday, April 22, 2009 5:34 PM
To: Noveck, Dave
Cc: Spencer Shepler; [email-masked]
Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

I thought that Benny had said
(http://www.ietf.org/mail-archive/web/nfsv4/current/msg06945.html)

-Jason

On Wed, Apr 22, 2009 at 5:16 PM, Noveck, Dave <[email-masked]>
wrote:

		Yes. I do suggest changing the requirement for the
client to set the layout hint from MUST to SHOULD and defining the
server's behavior in case the client did not set the layout hint to be
the same as currently defined for the case where the client set the
maximum io time via the layout hint to the "unbound" value.

		I agree with Benny. I would rather see this as SHOULD
with a description of what the server does when the client doesn't set
the attribute. This is all about tuning the grace period on the server,
and it seems easier to set a default on the server than to require every
block client. I'd rather see the protocol work without it, and then
allow clients to make the call in order to optimize the behavior during
the grace period.

	I think you, David Black and I have argued that the MUST is
right.  3-2 is not rough consensus but I suppose 4-1 is.  If anyone
moves from SHOULD to MUST, we are set.  If people decide that SHOULD is
right, I'm not going to be a holdout.  

		-----Original Message-----
		From: Jason Glasgow [mailto:[email-masked]] 
		Sent: Wednesday, April 22, 2009 11:05 AM
		To: Spencer Shepler; [email-masked]
		Cc: Noveck, Dave
		Subject: Re: [nfsv4] pNFS block layout & layout_hint
attribute

		Spencer,
		
		Let me attempt to clear up  the concerns.  The final
proposed text change is included below.  I believe that this
incorporates all the input from those who have posted on the issue.
		
		-Jason

		OLD:
		   The client MUST use SETATTR with a layout hint of
type
		   LAYOUT4_BLOCK_VOLUME to inform the server of its
maximum I/O time
		   prior to issuing the first LAYOUTGET operation.  The
maximum io time
		   hint is a per client attribute, and as such the
server SHOULD
		   maintain the value set by each client.
		
		NEW:
		   After a new client ID is established,
		   the client MUST use SETATTR with a layout hint of
type
		   LAYOUT4_BLOCK_VOLUME to inform the server of its
maximum I/O time
		   prior to issuing the first LAYOUTGET operation.
		   While the maximum io time
		   hint is a per file attribute, it is actually a
per-client,
		   characteristic. Thus the server MUST maintain the
last
		   maximum io time hint sent separately for each client.
Each time the
		   maximum io time changes, the server must apply it to
all files that
		   the client has a layout for.  If the client does not
		   specify this attribute on a file for which a block
layout is
		   requested, the server SHOULD use the most recent
value provided by
		
		   the same client for any file; if that client has
		   not provided a value for this attribute, the server
SHOULD
		   reject the layout request with the error
NFS4ERR_LAYOUTUNAVAILABLE.
		   The client SHOULD NOT send a SETATTR of the layout
hint with
		   every LAYOUTGET.

			I agree.

			> closure in an environment that is not hurried.

			My only issue is making sure that the attempt to
reach rough consensus
			and proceed with the alternate plan has a firm
time limit.  My proposal
			is the end of the week, defined as Midnight this
Friday in Jason's time
			zone.

			-----Original Message-----
			From: Spencer Shepler
[mailto:[email-masked]]
			Sent: Wednesday, April 22, 2009 1:38 AM
			To: Jason Glasgow
			Cc: [email-masked]
			Subject: Re: [nfsv4] pNFS block layout &
layout_hint attribute

			On Apr 10, 2009, at 10:31 AM, Jason Glasgow

			>    prior to issuing the first LAYOUTGET
operation.  The maximum io

			In re-reading this thread I am not sure I can
effectively discern the
			final, proposed text and bluntly don't know if
we have a rough
			consensus.
			
			With this note that all of the related NFSv4.1
I-Ds are ready and
			waiting in the editor's queue.
			
			Let's close on this quickly and move them
forward.   If we can't reach
			rough consensus, I will ask that the I-Ds be
moved forward as is and we
			start the bis process for the blocks work to
drive the details to
			closure in an environment that is not hurried.
			
			Spencer

---

From: Benny Halevy <[email-masked]>
Date: Thu, 23 Apr 2009 07:12:45 +0300

> either what he meant or what his position on "SHOULD vs. MUST" is now..

The "client MUST, server SHOULD" approach is reasonable and can work
in this case. Therefore, I have no objection to Jason's most recent proposal,
quoted down below.

Benny

> *Sent:* Wednesday, April 22, 2009 5:34 PM
> *To:* Noveck, Dave
> *Cc:* Spencer Shepler; [email-masked]
> On Wed, Apr 22, 2009 at 5:16 PM, Noveck, Dave <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>         the layout hint from MUST to SHOULD and defining the server's
>         behavior in case the client did not set the layout hint to be
>         the same as currently defined for the case where the client set
>         the maximum io time via the layout hint to the "unbound" value.
> 
>         description of what the server does when the client doesn't set
>         the attribute. This is all about tuning the grace period on the
>         server, and it seems easier to set a default on the server than
>         to require every block client. I'd rather see the protocol work
>         without it, and then allow clients to make the call in order to
>         optimize the behavior during the grace period.
> 
>         *From:* Jason Glasgow [mailto:[email-masked]
>         <mailto:[email-masked]>]
>         *To:* Spencer Shepler; [email-masked] <mailto:[email-masked]>
>         *Cc:* Noveck, Dave
>         text change is included below.  I believe that this incorporates
>         all the input from those who have posted on the issue.
> 
>         -Jason
> 
> 
>         OLD:
>         <[email-masked] <mailto:[email-masked]>> wrote:
> 
>             From: Spencer Shepler [mailto:[email-masked]
>             <mailto:[email-masked]>]
>             Cc: [email-masked] <mailto:[email-masked]>
>             maximum io

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 23 Apr 2009 08:15:42 -0400

That sounds like rough consensus to me.

Anybody disagree? 

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Thursday, April 23, 2009 12:13 AM
To: Noveck, Dave; Jason Glasgow
Cc: [email-masked]
Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

On Apr. 23, 2009, 3:24 +0300, "Noveck, Dave" <[email-masked]>
wrote:
> I read "Yup, that seems reasonable" as applying to the final paragaph 
> of David's message, rather than the entirety.  I guess the fact that 
> the singular was used led me to assume that only that one point was 
> being conceded.
>  

> either what he meant or what his position on "SHOULD vs. MUST" is
now..

The "client MUST, server SHOULD" approach is reasonable and can work in
this case. Therefore, I have no objection to Jason's most recent
proposal, quoted down below.

Benny

set

>            prior to issuing the first LAYOUTGET operation.  The
maximum
SHOULD
rough
>             consensus
>             > langauge from MUST to SHOULD, we open the door to
clients
>             repeatedly
>             > masking, and repeatedly getting error responses.  This
is
>             suboptimal
error
>             code to the
type
>             >    hint is a per client attribute, and as such the
server
type
per-client,
last
>             >    requested, the server SHOULD use the most recent
value
rough

---

From: "Everhart, Craig" <[email-masked]>
Date: Thu, 23 Apr 2009 09:07:37 -0400

I agree with Dave and David and, apparently, Jason.
 
        Craig

	From: Noveck, Dave 
	Sent: Wednesday, April 22, 2009 5:17 PM
	To: Jason Glasgow; Spencer Shepler; [email-masked]
	Subject: Re: [nfsv4] pNFS block layout & layout_hint attribute

	 

		-----Original Message-----
		From: Jason Glasgow [mailto:[email-masked]] 
		Sent: Wednesday, April 22, 2009 11:05 AM
		To: Spencer Shepler; [email-masked]
		Cc: Noveck, Dave
		Subject: Re: [nfsv4] pNFS block layout & layout_hint
attribute

		Spencer,

		-Jason

		OLD:
		
		NEW:

		

			> Let's close on this quickly and move them
forward.

			I agree.

			-----Original Message-----
			From: Spencer Shepler

			
			
			>    prior to issuing the first LAYOUTGET
operation.

			

			Spencer

---

From: Spencer Shepler <[email-masked]>
Date: Fri, 24 Apr 2009 00:26:36 -0500

Thanks to all for the input.

The draft-ietf-nfsv4-pnfs-block-12.txt I-D will be updated
during the "authors 48 hours" with the "NEW" text below.

Spencer