Subject: [nfsv4] Layout Structures Improvement

From: "Anatoly Pinchuk" <[email-masked]>
Date: Fri, 15 Feb 2008 17:56:26 -0800

Some layout related structures of the current draft are not optimal and
bring unnecessary complications into the protocol. With this posting I
propose simple solution of the problem.

THE PROBLEM

Simple LAYOUTGET semantic which existed in draft 15 is broken. According
to it a client requests layout for a range, the server returns layout
for a sub-range. Layout has only one type and one IO mode. Now the
client gets array of sub-layouts (ranges, IO modes, types). Also there
are dependencies between the sub-layouts. Since addition of the array in
draft 16 targeted specific file layout type related problem, it can and
should be done at file layout level without disturbing generic LAYOUTGET
semantic.

Unnecessary field duplications can lead to rather long result of
LAYOUTGET. For example, N * (sizeof(offset4) + sizeof(length4) +
sizeof(layoutiomode4) + sizeof(layouttype4) + sizeof(offset4)) bytes are
needed to carry result offsets, lengths, IO modes, and layout types,
where N is a number of the layout4 structures in LAYOUTGET4resok. With
the fix we have the same information can be passed using just
sizeof(offset4) + sizeof(length4) + sizeof(layoutiomode4) +
sizeof(layouttype4) + N * sizeof(offset4) bytes. In particular, such
difference in length could be decisive since the client controls length
of the result with loga_maxcount field.

The field duplication above complicates the protocol and, therefore, a
client and server implementation. Such sentences as "Each element of
logr_layout MUST have the same iomode" will not be needed if we
structure result of LAYOUTGET right (just one iomode field). It is also
not obvious why the protocol defines several logr_layout elements with
fields having the same value guaranteed.

THE SOLUTION

A link to the solution is
http://www.ietf.org/mail-archive/web/nfsv4/current/msg05174.html

DIFF OF THE SOLUTION AGAINST DOT-X FILE

-%/* Encoded in the loc_body field of type layout_content4: */

-struct nfsv4_1_file_layout4 {

-         deviceid4      nfl_deviceid;

-         nfl_util4      nfl_util;

-         uint32_t       nfl_first_stripe_index;

-         offset4        nfl_pattern_offset;

-         nfs_fh4        nfl_fh_list<>;

+%/* Segment - data distribution descriptor of a client file range */

+struct nfsv4_1_file_layout_segment4 {

+         offset4        nfls_off;

+         deviceid4      nfls_deviceid;

+         nfl_util4      nfls_util;

+         uint32_t       nfls_first_stripe_index;

+         nfs_fh4        nfls_fh_list<>;

 };

 %

+%/* Encoded in the loc_body field of type layout_content4: */

+typedef nfsv4_1_file_layout_segment4 nfsv4_1_file_layout4<>;

+%

 %/*

 % * Encoded in the lou_body field of type layoutupdate4:

 % *      Nothing. lou_body is a zero length array of octets.

@@ -2317,7 +2322,7 @@

 struct LAYOUTGET4resok {

         bool               logr_return_on_close;

         stateid4           logr_stateid;

-        layout4            logr_layout<>;

+        layout4            logr_layout;

 };

- Anatoly

_______________________________________________
nfsv4 mailing list
[email-masked]
http://www.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Sat, 16 Feb 2008 08:23:41 -0500

> Layout has only one type and one IO mode. Now the client gets array of
sub-layouts (ranges, IO modes, types).
 
The types of all the layouts MUST be equal to the requested type.  That
isn't in draft-19, but it has been discussed and agreed to by everyone
and I posted the changes and have checked them in.  Since all the types
are equal to the requested type, they must all be equal to one another.
 
And the requirement that the type must equal the requested type is
independent of whether we decide on a single-layout or multiple-layout
model.
 
The important equality is between the layout and the requested type, and
while there has been discussion of eliminating the type in some cases,
the working group has decided that space optimality is not the
controlling consideration here.
 
I have not done exhaustive study of the pros and cons of this, but even
if there are no cons at all, it seems that the pros are kind of thin,
given where we are.  The simplification of eliminating one constraint
(on iomode) on the returned layouts is not large.  Also note, that if
your layout type only returns single layouts, the constraint fades away.
 
> In particular, such difference in length could be decisive since the
client controls length of the result with loga_maxcount field.
 
It could be decisive for a particular request with an inappropriately
low value of loga_maxcount, but the size requirements are clear as you
have explained.  Also note here that the server has the option of
returning fewer layouts.
 
I can see how this approach might seem an ugly blemish to you, but I
just don't see that the problems you have outlined are anything more
than esthetic ones.  My vote is to keep things as they are.

________________________________

From: Anatoly Pinchuk [mailto:[email-masked]] 
Sent: Friday, February 15, 2008 8:56 PM
To: [email-masked]
Subject: [nfsv4] Layout Structures Improvement

THE PROBLEM

THE SOLUTION

-         nfl_util4      nfl_util;

-         nfs_fh4        nfl_fh_list<>;

+         offset4        nfls_off;

+         nfl_util4      nfls_util;

 };

 %

+%

 %/*

@@ -2317,7 +2322,7 @@

 struct LAYOUTGET4resok {

         stateid4           logr_stateid;

-        layout4            logr_layout<>;

+        layout4            logr_layout;

 };

- Anatoly

---

From: Robert Gordon <[email-masked]>
Date: Sat, 16 Feb 2008 07:44:31 -0600

On Feb 16, 2008, at 7:23 AM, Noveck, Dave wrote:

> My vote is to keep things as they are.

Agreed.

Robert.

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Tue, 19 Feb 2008 10:55:53 -0800

I believe that a vote should be based on understanding and analysis of a
proposed solution. The following fragment indicates that your
misinterpreted some parts of the proposal.

----------------

----------------

My proposal has nothing to do with value of layout type in relation to
the requested type. Also, it is not about eliminating one constraint (on
iomode).

Without reiteration of the solution itself, I would like to repeat its
benefits. 

- it makes sure that semantic of LAYOUTGET (one range, one IO mode, one
layout type) does correspond to its syntax (one layout4 element defining
range, IO mode, and layout type; not array of them)

- introduces right layering: (common layout) - (file layout type) and
solves file layout type related problems at file layout layer

- since unneeded array of layout4 and element field dependencies are
removed from the protocol, the protocol and client/server
implementations become simpler. So, there will be no need to
describe/set/validate dependencies between loc_type, lo_offset,
lo_length, and lo_iomode fields of the different elements of logr_layout
array. For example, such sentences as "Each element of logr_layout MUST
have the same iomode" will not be needed anymore.

- shortens result of LAUOUTGET for file layouts based on more than one
segment

Because you voted against the proposed solution, one can guess that you
found some downsides in it. But your reply has none. It is not really a
downside to have an esthetic solution or difference in length "decisive
for a particular request". May be you can explain why you do not want to
match the protocol semantic with syntax, have right layering, simplify
the protocol and server/client implementations, and shorten result of
LAUOUTGET (see list above). Is there something in that list you consider
as a downside? Are there any downsides in the solution?

It seems that you use the reference to were we are as justification for
not making changes. Actually, reality calls for completely opposite - to
find problems in the protocol and fix them as soon as possible. We are
at the protocol, especially its pNFS part, reviewing stage. The stage
supposes that we find and fix protocol issues before they became the
standard. That is exactly what I am trying to achieve.

- Anatoly

From: Noveck, Dave [mailto:[email-masked]] 
Sent: Saturday, February 16, 2008 5:24 AM
To: Anatoly Pinchuk; [email-masked]
Subject: RE: [nfsv4] Layout Structures Improvement

THE PROBLEM

THE SOLUTION

-         nfl_util4      nfl_util;

-         nfs_fh4        nfl_fh_list<>;

+         offset4        nfls_off;

+         nfl_util4      nfls_util;

 };

 %

+%

 %/*

@@ -2317,7 +2322,7 @@

 struct LAYOUTGET4resok {

         stateid4           logr_stateid;

-        layout4            logr_layout<>;

+        layout4            logr_layout;

 };

- Anatoly

---

From: Spencer Shepler <[email-masked]>
Date: Tue, 19 Feb 2008 15:07:47 -0600

On Feb 19, 2008, at 12:55 PM, Anatoly Pinchuk wrote:

> I believe that a vote should be based on understanding and analysis  
> of a proposed solution. The following fragment indicates that your  

Note that there is not going to be a "vote"; a demonstration
of rough consensus is needed to enact changes.

Spencer
>
>
> ----------------
>
> The types of all the layouts MUST be equal to the requested type.   
> That isn't in draft-19, but it has been discussed and agreed to by  
> everyone and I posted the changes and have checked them in.  Since  
> all the types are equal to the requested type, they must all be  
> equal to one another.
>
>
>
> independent of whether we decide on a single-layout or multiple- 
> layout model.
>
>
>
> The important equality is between the layout and the requested  
> type, and while there has been discussion of eliminating the type  
> in some cases, the working group has decided that space optimality  
> is not the controlling consideration here.
>
>
>
> I have not done exhaustive study of the pros and cons of this, but  
> even if there are no cons at all, it seems that the pros are kind  
> of thin, given where we are.  The simplification of eliminating one  
> constraint (on iomode) on the returned layouts is not large.  Also  
> note, that if your layout type only returns single layouts, the  
> constraint fades away.
>
> ----------------
>
> My proposal has nothing to do with value of layout type in relation  
> to the requested type. Also, it is not about eliminating one  
> constraint (on iomode).
>
>
>
> Without reiteration of the solution itself, I would like to repeat  
> its benefits.
>
> - it makes sure that semantic of LAYOUTGET (one range, one IO mode,  
> one layout type) does correspond to its syntax (one layout4 element  
> defining range, IO mode, and layout type; not array of them)
>
> - introduces right layering: (common layout) - (file layout type)  
> and solves file layout type related problems at file layout layer
>
> - since unneeded array of layout4 and element field dependencies  
> are removed from the protocol, the protocol and client/server  
> lo_length, and lo_iomode fields of the different elements of  
> logr_layout array. For example, such sentences as “Each element of  
> logr_layout MUST have the same iomode” will not be needed anymore.
>
> - shortens result of LAUOUTGET for file layouts based on more than  
> one segment
>
>
>
> Because you voted against the proposed solution, one can guess that  
> you found some downsides in it. But your reply has none. It is not  
> really a downside to have an esthetic solution or difference in  
> length "decisive for a particular request". May be you can explain  
> why you do not want to match the protocol semantic with syntax,  
> have right layering, simplify the protocol and server/client  
> implementations, and shorten result of LAUOUTGET (see list above).  
> Is there something in that list you consider as a downside? Are  
> there any downsides in the solution?
>
>
>
> It seems that you use the reference to were we are as justification  
> for not making changes. Actually, reality calls for completely  
> opposite – to find problems in the protocol and fix them as soon as  
> possible. We are at the protocol, especially its pNFS part,  
> reviewing stage. The stage supposes that we find and fix protocol  
> issues before they became the standard. That is exactly what I am  
> trying to achieve.
>
>
>
> - Anatoly
>
>
>
>
>
> > Layout has only one type and one IO mode. Now the client gets  
> array of sub-layouts (ranges, IO modes, types).
>
>
>
> The types of all the layouts MUST be equal to the requested type.   
> That isn't in draft-19, but it has been discussed and agreed to by  
> everyone and I posted the changes and have checked them in.  Since  
> all the types are equal to the requested type, they must all be  
> equal to one another.
>
>
>
> independent of whether we decide on a single-layout or multiple- 
> layout model.
>
>
>
> The important equality is between the layout and the requested  
> type, and while there has been discussion of eliminating the type  
> in some cases, the working group has decided that space optimality  
> is not the controlling consideration here.
>
>
>
> I have not done exhaustive study of the pros and cons of this, but  
> even if there are no cons at all, it seems that the pros are kind  
> of thin, given where we are.  The simplification of eliminating one  
> constraint (on iomode) on the returned layouts is not large.  Also  
> note, that if your layout type only returns single layouts, the  
> constraint fades away.
>
>
>
> > In particular, such difference in length could be decisive since  
> the client controls length of the result with loga_maxcount field.
>
>
>
> It could be decisive for a particular request with an  
> inappropriately low value of loga_maxcount, but the size  
> requirements are clear as you have explained.  Also note here that  
> the server has the option of returning fewer layouts.
>
>
>
> I can see how this approach might seem an ugly blemish to you, but  
> I just don't see that the problems you have outlined are anything  
> more than esthetic ones.  My vote is to keep things as they are.
>
>
>
> Some layout related structures of the current draft are not optimal  
> and bring unnecessary complications into the protocol. With this  
> posting I propose simple solution of the problem.
>
>
>
> THE PROBLEM
>
> Simple LAYOUTGET semantic which existed in draft 15 is broken.  
> According to it a client requests layout for a range, the server  
> returns layout for a sub-range. Layout has only one type and one IO  
> mode. Now the client gets array of sub-layouts (ranges, IO modes,  
> types). Also there are dependencies between the sub-layouts. Since  
> addition of the array in draft 16 targeted specific file layout  
> type related problem, it can and should be done at file layout  
> level without disturbing generic LAYOUTGET semantic.
>
>
>
> sizeof(layoutiomode4) + sizeof(layouttype4) + sizeof(offset4))  
> bytes are needed to carry result offsets, lengths, IO modes, and  
> layout types, where N is a number of the layout4 structures in  
> LAYOUTGET4resok. With the fix we have the same information can be  
> passed using just sizeof(offset4) + sizeof(length4) + sizeof 
> (layoutiomode4) + sizeof(layouttype4) + N * sizeof(offset4) bytes.  
> In particular, such difference in length could be decisive since  
> the client controls length of the result with loga_maxcount field.
>
>
>
> The field duplication above complicates the protocol and,  
> therefore, a client and server implementation. Such sentences as  
> “Each element of logr_layout MUST have the same iomode” will not be  
> needed if we structure result of LAYOUTGET right (just one iomode  
> field). It is also not obvious why the protocol defines several  
> logr_layout elements with fields having the same value guaranteed.
>
>
>
> THE SOLUTION
>
> A link to the solution is http://www.ietf.org/mail-archive/web/ 
> nfsv4/current/msg05174.html
>
>

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 19 Feb 2008 16:18:52 -0500

> My vote is to keep things as they are. 

Mea culpa.

I meant "My view of the matter is that things are better left as they
are."

-----Original Message-----
From: Spencer Shepler [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2008 4:08 PM
To: Anatoly Pinchuk
Cc: Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] Layout Structures Improvement

On Feb 19, 2008, at 12:55 PM, Anatoly Pinchuk wrote:

> I believe that a vote should be based on understanding and analysis of

> a proposed solution. The following fragment indicates that your 

Note that there is not going to be a "vote"; a demonstration of rough
consensus is needed to enact changes.

Spencer
> everyone and I posted the changes and have checked them in.  Since all

> the types are equal to the requested type, they must all be equal to 
> one another.
>
>
>
> The important equality is between the layout and the requested type, 
> and while there has been discussion of eliminating the type in some 
> cases, the working group has decided that space optimality is not the 
> even if there are no cons at all, it seems that the pros are kind of 
> thin, given where we are.  The simplification of eliminating one 

> the requested type. Also, it is not about eliminating one constraint 
> (on iomode).
>
>
>
> logr_layout array. For example, such sentences as "Each element of  
> logr_layout MUST have the same iomode" will not be needed anymore.
>
> opposite - to find problems in the protocol and fix them as soon as  
> "Each element of logr_layout MUST have the same iomode" will not be

---

From: "Noveck, Dave" <[email-masked]>
Date: Tue, 19 Feb 2008 18:22:42 -0500

> I believe that a vote should be based on understanding and 
> analysis of a proposed solution.
 
Spencer informs us it is not a vote but merely my view based on my
understanding and analysis of the proposed solution.  I'm open to the
possibility that my understanding and analysis is not correct but so far
I have not seen any indication that that is so.
 
> My proposal has nothing to do with value of layout 
> type in relation to the requested type. Also, it 
> is not about eliminating  one constraint (on iomode).
 
You said:
 
> Layout has only one type and one IO mode. Now 
> the client gets array of sub-layouts (ranges, 
> IO modes, types). 

and:

> The field duplication above complicates the 
> protocol and, therefore, a client and server 
> Implementation. Such sentences as "Each 
> element of logr_layout MUST have the same 
> iomode" will not be needed if we structure 
> result of LAYOUTGET right (just one iomode 
> field). It is also not obvious why the protocol 
> defines several logr_layout elements with fields 
> having the same value guaranteed.

It sounds to me like you are concerned with such constraints/sentences
as one regarding the iomodes being the same.  The one regarding the
types being the same is similar.

I think it is a relevant point that that latter constraint is really
just the requirement that the type returned has to be the type requested
and so there is no additional constraint here.

The example of the constraint on the type is relevant.  Implementation
of the protocol involves numerous constraints regarding matching fields
on requests and replies and some internal to the reply.  I don't that it
is any big deal or that it poses any noticeable difficulty for the
client and server implementation.  My response to the claim that this
makes things difficult for the implementation is "deal with it".  It is
just is not any big deal.

> Because you voted against the proposed 
> solution, one can guess that you found 
> some downsides in it.

The downside is that it is a change without anything to justify it.  If
it is the "solution", what is the problem?  You do have explanations of
why you would have done things differently and I have not really
considered whether I would agree with you considering the matter on a
clean-slate basis.  But we don't have a clean slate here, we have a
proposed protocol and are seeing if there are substantial problems that
would require change.  My approach is to put the burden on those
proposing a change to justify the change and I would expect that
justification to include either a statement of the existing approach
being unworkable for somebody or a substantial performance issue.  Your
issues don't seem to me to be even close to that.

> Actually, reality calls for completely 
> opposite - to find problems in the protocol 
> and fix them as soon as possible. 

In my view, only if they reach a certain level of significance.  This is
a subjective judgment, but I will give you my subjective judgment.
Other members of the working group are free to give you their subjective
judgment applied to their standards in this regard.

> We are at the protocol, especially its pNFS 
> part, reviewing stage. The stage supposes that 
> we find and fix protocol issues before they 
> became the standard. That is exactly what I am 
> trying to achieve.

I appreciate that, but I believe that if we try to eliminate too large a
class of protocol issues, we will never get finished.  My feeling is
that draft-19 is pretty good (I admit I am biased) and that until
someone shows me a bigger issue than you have, we should avoid making
significant changes to the protocol.

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Wed, 20 Feb 2008 10:56:16 -0800

Let us focus on your analysis. According to it there is a downside in
the proposal:

--
The downside is that it is a change without anything to justify it.
--

This is list of benefits (changes) the solution brings

Do you mean that matching the protocol semantic with syntax, having
right layering, simplifying the protocol and server/client
implementations, and shortening result of LAYOUTGET (see list above) do
not justify the change?

- Anatoly

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 20 Feb 2008 14:28:22 -0500

> implementations, and shortening result of LAYOUTGET (see list above)
do

Obviously so, but my opinion is only that of a single member of group.
If there is a rough consensus, that this does justify making a change,
then we'll do that.  This is a case in which we look at the same set of
facts and come to different judgments.  It doesn't seem to me that
further discussion between us is going to change that. 

-----Original Message-----
From: Anatoly Pinchuk [mailto:[email-masked]] 
Sent: Wednesday, February 20, 2008 1:56 PM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Layout Structures Improvement

- Anatoly

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Wed, 20 Feb 2008 11:41:54 -0800

I did not start voting and am OK if a proposed change is accepted or
denied depending on result of the change analysis.

-Anatoly

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Tuesday, February 19, 2008 1:08 PM
To: Anatoly Pinchuk
Cc: Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] Layout Structures Improvement

Spencer

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Thu, 21 Feb 2008 11:59:56 -0800

I am not trying to change your judgment. Obviously it is up to you to
keep or change your initial opinion. My goal in the discussion is to
make sure that we discuss what I proposed, not something else. Also I
would like to emphasize the benefits, invite you to comment on the
benefits and provide downsides, if any. There was no disagreement on the
list of benefits and no downsides provided. The only "downside" you
found is "The downside is that it is a change without anything to
justify it". This is not a proposal downside since justification is not
part of the proposal at all. This is your opinion about the proposal. I
think we are done with technical part of the discussion.

The summary of the analysis and follow-up discussion is
- Change I propose improves the protocol and simplifies implementation 
- It does not have any downsides

Some non-technical questions still remain though. Answers are important
as they affect a way one can improve the protocol. As you noted, our
conclusions from the same set of facts are different. Since I believe
that we approach all proposed changes in the same manner, we follow
different policies on acceptance of the changes. My policy is that if a
proposed change improves the protocol and does not have downsides, it
should be incorporated into the protocol. It is unclear to me what your
policy is. Can you explain it?

- Anatoly

-----Original Message-----
From: Anatoly Pinchuk [mailto:[email-masked]] 
Sent: Wednesday, February 20, 2008 1:56 PM
To: Noveck, Dave; [email-masked]
Subject: RE: [nfsv4] Layout Structures Improvement

- Anatoly

---

From: "J. Bruce Fields" <[email-masked]>
Date: Thu, 21 Feb 2008 17:35:33 -0500

On Thu, Feb 21, 2008 at 11:59:56AM -0800, Anatoly Pinchuk wrote:
> My policy is that if a proposed change improves the protocol and does
> not have downsides, it should be incorporated into the protocol. It is
> unclear to me what your policy is. Can you explain it?

I assume the goal is to get high-quality pfns implementations out there
and solving problems for people as soon as possible.

Since there is some ongoing implementation work right now, any change to
the protocol may delay that work by the amount of time required for
everyone to adapt to the change.

One way to put it might be: is a given improvement going to get good
implementations to people sooner or later?

So we still seem to be sneaking in some large-ish rewrites of text,
because such delays are rather low in that case.  (Nobody has to modify
their implementation to take into them into account.)  But on-the-wire
changes get more scrutiny.

Personally it seems to me that the existing implementations are all
young enough that simple mechanical xdr changes that make the protocol
simpler might be a net win.  But, whatever--the point is that it's not
purely a matter of whether a given change is an improvement or not.

--b.

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Thu, 21 Feb 2008 22:20:17 -0800

I am trying to improve the protocol, so people can get high quality
implementations of high quality protocol.

I would like to clarify my stance on the policy. I said an improvement
"should", not must, be incorporated. What I mean is that unless there is
a dire reason not to incorporate an improvement into the protocol, it
needs to be incorporated. 

Since we are in the review phase, changes are flowing into the protocol
anyway. Even text ones, updating protocol semantics, might require
implementation modifications. Also, as you noted, implementations are
young. Some of them have not even started yet. We do not have a standard
after all. Because of the above, it is hard to imagine a reason
preventing an improvement adoption. It would be more difficult to
improve the protocol later.

- Anatoly

-----Original Message-----
From: J. Bruce Fields [mailto:[email-masked]] 
Sent: Thursday, February 21, 2008 2:36 PM
To: Anatoly Pinchuk
Cc: Noveck, Dave; [email-masked]
Subject: Re: [nfsv4] Layout Structures Improvement

--b.