Subject: [nfsv4] File Layout Questions (Layouts of the same file and Packing)

From: "Anatoly Pinchuk" <[email-masked]>
Date: Thu, 1 Nov 2007 15:38:03 -0700

Thanks for the replies on the layout range subject.

There are two more topics I would like to clarify

1. Layouts of the same client file

There seems to be an agreement that more than one layout is possible
corresponding to a given client file. If so, it would be great to
describe if there are dependencies between the layouts of the same file.
One interesting question is if the stripe unit size can be different in
such layouts.

2. Sparse or dense packing

Currently packing is stored in a layout which makes it impossible to
bring two data servers with different packing implementations under the
same layout. Also, it is unlikely that a data server implements both
sparse and dense packing. Would it be more flexible then to have packing
defined per equivalent data server group or even per data server?

Regards,

Anatoly

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Mike Eisler <[email-masked]>
Date: Thu, 1 Nov 2007 18:49:50 -0700 (PDT)

> One interesting question is if the stripe unit size can be different

The spec doesn't disallow this. If you want to suggest some
test to make this clear, that would be welcome.

> defined per equivalent data server group or even per data server?

I thought the packing and stripe unit size should be in the device
address, and asked this question of WG several on June 12, 2007.
The consensus was to define it in the layout. So that's where
we are.

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Mon, 5 Nov 2007 14:48:05 -0800

1. If the client file layouts can have different stripe unit sizes we need more than just a text clarification. In that case the layout structure should include additional information to calculate a data file offset for dense packing. Offset returned in layout4 structure and stripe unit size is not enough to do the calculation. Here is an example:
 
There are devices:
struct nfsv4_1_file_layout_ds_addr4
dev0 = { // ID is 0
    {0},
    {{A}}
},
dev1 = { // ID is 1
    {0},
    {{B}}
};
 
File Layouts for a client file
struct nfsv4_1_file_layout4
fl0 = {
    0,
    x, // stripe unit size of 0x400 is coded with x
    0,
    fh0
},
fl1 = {
    1,
    y, // stripe unit size of 0x1000 and dense packing are coded with y
    0,
    fh1
};
 
fl0 and fl1 correspond to file ranges [0, 0x400-1] and [0x400, 0x1400-1];
 
Now a client requests layout for an offset of 0x800 and length 0x400 and the reply is:
 
struct layout4
lo = {
    0x800,
    0x400,
    lo_iomode, // some IO mode
    lo_content // see below
};
 
struct layout_content4
lo_content = {
    LAYOUT4_NFSV4_1_FILES,
    loc_body // see below
};
loc_body contains fl1 defined above.
 
So, the client knows the server (B) and the file handle (fh1) to do IO on. Offset for the fh1 is needed as well. The formula from paragraph 14.5 does not provide correct offset since
 
data_file_offset = floor(file_offset / stripe_width)
* stripe_unit_size + file_offset % stripe_unit_size =
floor(0x800/0x1000) * 0x1000 + 0x800 % 0x1000 = 0x800;
 
and the correct offset is

rel_off = (file_offset - layout_offset) = (0x800 - 0x400) = 0x400; 
data_file_offset = floor(rel_offset / stripe_width)
* stripe_unit_size + rel_offset % stripe_unit_size =
floor(0x400/0x1000) * 0x1000 + 0x400 % 0x1000 = 0x400;
 
Offset of first byte the layout is applicable to (layout_offset = 0x400) is used in the calculation of the data file offset. Therefore, it should be stored in the nfsv4_1_file_layout4 structure returned to the client.

2. Packing can be kept in the layout, but doing so limits number of ways the layouts can be created. Pathological example would be having just two data servers A and B with dense and sparse packing implemented respectively. Then no layout can correspond to both A and B data file servers. In such configuration a multi-path can not be created and client file data access parallelism can not be achieved at all. A remedy could be storing the packing per data server. As for the stripe unit size, since it describes the data distribution, it probably should follow stripe indices and be stored in a device structure. In general, packing and stripe unit size rather belong to the device than layout.

---

From: Mike Eisler <[email-masked]>
Date: Tue, 13 Nov 2007 20:43:30 -0800 (PST)

You example seems to request a layout for a range that overlaps two
striping patterns, and it returns a layout that overlaps, which I
think is a server defect.

To resolve a related another issue, 
draft 16 now has LAYOUTGET return an array
of layout4 data types:

layout4            logr_layout<>;

Does that resolve the issue? At least it allows the server to
do something saner.

--- Anatoly Pinchuk <[email-masked]> wrote:

> 1. If the client file layouts can have different stripe unit sizes we
> need more than just a text clarification. In that case the layout
> structure should include additional information to calculate a data
> file offset for dense packing. Offset returned in layout4 structure
> and stripe unit size is not enough to do the calculation. Here is an
> example:
>  
> There are devices:
> So, the client knows the server (B) and the file handle (fh1) to do
> IO on. Offset for the fh1 is needed as well. The formula from
> paragraph 14.5 does not provide correct offset since
>  
> Offset of first byte the layout is applicable to (layout_offset =
> 0x400) is used in the calculation of the data file offset. Therefore,
> it should be stored in the nfsv4_1_file_layout4 structure returned to
> the client.
> 
> 2. Packing can be kept in the layout, but doing so limits number of
> ways the layouts can be created. Pathological example would be having
> just two data servers A and B with dense and sparse packing
> implemented respectively. Then no layout can correspond to both A and
> B data file servers. In such configuration a multi-path can not be
> created and client file data access parallelism can not be achieved
> at all. A remedy could be storing the packing per data server. As for
> the stripe unit size, since it describes the data distribution, it
> probably should follow stripe indices and be stored in a device
> structure. In general, packing and stripe unit size rather belong to
> the device than layout.
> 
> 
> 
> > One interesting question is if the stripe unit size can be
> different
> > in
> > such layouts.
>

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Wed, 14 Nov 2007 11:13:40 -0800

Actually, in the example there are two striping patterns (dev0 and dev1)
but
returned layout (lo) does not cross the device boundaries. It is solely 
based on the device dev1 and, therefore, does not overlap.

Having array of layout4 as return result is not going to help to resolve
the
issue since in the example just one layout corresponds to the client
provided offset (0x800) and length (0x400).

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Tuesday, November 13, 2007 8:44 PM
To: Anatoly Pinchuk; [email-masked]; [email-masked]
Subject: RE: [nfsv4] File Layout Questions (Layouts of the same file and
Packing)

layout4            logr_layout<>;

---

From: Mike Eisler <[email-masked]>
Date: Wed, 14 Nov 2007 20:07:16 -0800 (PST)

> > struct nfsv4_1_file_layout4

deviceID 0

> >     x, // stripe unit size of 0x400 is coded with x

deviceID 1
> >     y, // stripe unit size of 0x1000 and dense packing are coded
> with
> > y
> >     0,
> >     fh1
> > };
> >  
> > Now a client requests layout for an offset of 0x800 and length
> 0x400
> > and the reply is:
> >  
> > struct layout4
> > lo = {
> >     0x800,

offset 0x800

> > floor(0x400/0x1000) * 0x1000 + 0x400 % 0x1000 = 0x400;

Unless fh1 and fh0 are the same.

This is a big problem and I don't have an immediate
answer other than to suggest we remove dense
packing from the protocol and rely on data server intelligence
to pack stripe units.

---

From: Mike Eisler <[email-masked]>
Date: Wed, 14 Nov 2007 21:19:25 -0800 (PST)

A majority of the editors agree we should remove DENSE packing from
the NFSv4.1 pNFS file layout specification since as Anatoly points out,
it doesn't work.

I plan to start ripping it out in a few hours unless someone can
convince us otherwise.

--- Mike Eisler <[email-masked]> wrote:

> > > 1. If the client file layouts can have different stripe unit
> sizes
> > we

---

From: "Muntz, Daniel" <[email-masked]>
Date: Thu, 15 Nov 2007 00:15:03 -0800

Would it be correct to say that this decision is being made under these
assumptions:

1) dense striping is less valuable than mixed stripe sizes
2) do not want to provide an option to choose either mixed stripe sizes
_OR_ dense stripes per [file/fs/...]
3) mixed stripe sizes are believed to be supportable except for this
conflict with dense striping (I'm still trying to convince myself on
this point).

  -Dan

> Sent: Wednesday, November 14, 2007 9:19 PM
> To: [email-masked]
> Subject: [nfsv4] pNFS Dense Packing Problem
> 
> > > > and stripe unit size is not enough to do the 
> calculation. Here is
> > > an
> > > > example:
> > > >  
> > > > There are devices:

---

From: Benny Halevy <[email-masked]>
Date: Thu, 15 Nov 2007 11:45:16 +0200

The initial assumption AFAICT was that there is just one stripe (unit) size
for the file and the data_file_offset calculation was based on that.
This can be fixed by either allowing mixed stripe unit sizes and correcting
the math in section 13.4.2.  Interpreting the File Layout Using Dense Packing,
or by eliminating mixed stripe sizes, e.g. by moving the stripe unit size
to nfsv4_1_file_layouthint4, along side with nflh_stripe_count.  BTW, if you
allow mixed stripe sizes why don't you provide for mixed stripe counts?
In CMU's model the file could be comprised of a concatenation of "patterns"
each with it's own striping geometry, and frankly, intuitively this makes
more sense to me than a set stripe count and variable stripe unit size...
Anyhow, I'm not sure how mixed stripe unit sizes work with sparse
packing and what's their use case. If supporting them isn't justified
I'd rather eliminate that feature rather than dense packing.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Thu, 15 Nov 2007 04:30:26 -0800 (PST)

> conflict with dense striping (I'm still trying to convince myself on

Dense packing and mixed stripe sizes are suppportable as
explicit entries in the protocol. It requires
making the specification text more complex, hence easier to
misinterpret.  

Why do we have dense packing?

1. Some filesystems can't do holes well.

2. readahead I/O is harder to detect

Dense packing can be done without making it part of the
file layout: just implement data servers to map the sparse offsets
to dense offsets.

> > > > > structure should include additional information to calculate
> a
> > > data
> > > > > fl0 and fl1 correspond to file ranges [0, 0x400-1] and
> [0x400,
> > > > > 0x1400-1];
> > > > >  
> > > > > So, the client knows the server (B) and the file handle (fh1)
> to
> > > do

---

From: Mike Eisler <[email-masked]>
Date: Thu, 15 Nov 2007 04:46:16 -0800 (PST)

> for the file and the data_file_offset calculation was based on that.

Apparently.

> to nfsv4_1_file_layouthint4, along side with nflh_stripe_count.  BTW,

The stripe unit size can't be in the hint, because the hint
is write only.

> if you

We do provide mixed stripe counts.

``The number of elements in nflda_stripe_indices is always equal to the
stripe count.''

We can have multiple instances data types of type nfsv4_1_file_layout4
in a layout, and there is one nflda_stripe_indices field per instance
of
nfsv4_1_file_layout4.

> each with it's own striping geometry, and frankly, intuitively this

Which is what we have today.

> makes

As long as the array of elements of type layout4 don't overlap,
I believe mixed stripe unit sizes work fine for sparse packing.

Anyway, I never wanted mixed stripe unit sizes; the consensus was
to have them. Given that consensus, I'm saying dense packing has to
be deleted.

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Thu, 15 Nov 2007 12:06:10 -0800

I think answers to the following questions might help to decide on a
fate of dense packing.

- Do we need dense packing at all? 
- If answer is yes, why would we prefer a client to do the offset
translation? 
- If there is meaning to have the translation in the client, is it
possible (difficult) to change the layout structure to make such
translation possible? 
 
It seems that enough rationale was provided to confirm that dense
packing is a valuable feature. Also, there is a reason to keep the
translation inside the client since it will offload the server. As for
the layout structure, a relative offset field can be added to it and,
therefore, data file offset can be calculated.

-Anatoly

> From: Mike Eisler [mailto:email2mre-ietf at yahoo.com] 
> To: nfsv4 at ietf.org

---

From: "Muntz, Daniel" <[email-masked]>
Date: Thu, 15 Nov 2007 13:46:39 -0800

Who wants multiple stripe unit sizes and what are they doing with them?

One thing that comes to mind is perhaps mixed stripe sizes are being
used as an analogue to extents in physical file systems (or perhaps to
directly map to extents, or similar structures, in an underlying
physical fs on the DS).

Other mixed stripe size applications could be implemented as a fixed
"base" stripe size, and an appropriate application of the layout
mechanism to create stripe groups to emulate mixed stripe sizes (with
the restriction that the possible "stripe sizes" are multiples of the
"base stripe size").

I get the impression that the multiple stripe size stuff is, to quote
Eisler on an unrelated topic, "1/4 baked."  If including them has
provably broken a relatively simple part of the pNFS protocol, what else
are we going to hit in more complex areas (locking, restriping,
replicas, ...)?  Are there white-papers, or research demonstrating the
benefit outweighs the complexity and overhead introduced?  Is this a
90/10 situation (or a 99.999/.001) where a huge percentage of users
would be just as well off with a fixed stripe size?

W.r.t. dense packing, even _if_ the fs gets the hole-making correct in
the first place, there are always opportunities for holes to become
allocated disk blocks.

Perhaps support for multiple stripe sizes could be OPTIONAL on the
client.  It's an ugly thought, but then the market can decide.  Clients
that don't support them can read/write through the MDS, or the server
could hide multiple stripe sizes from these clients (now _there's_ some
added complexity on the server side... :-)

  -Dan

-----Original Message-----
From: Anatoly Pinchuk [mailto:[email-masked]] 
Sent: Thursday, November 15, 2007 12:06 PM
To: [email-masked]
Subject: RE: [nfsv4] pNFS Dense Packing Problem

 

-Anatoly

> the NFSv4.1 pNFS file layout specification since as Anatoly points 
> out, it doesn't work.
>

---

From: "Anatoly Pinchuk" <[email-masked]>
Date: Thu, 15 Nov 2007 14:22:56 -0800

Multiple stripe unit sizes can be utilized in some server
implementations. Also, probably, it is not so difficult to support.

Real problem though is not multiple stripe unit sizes (I just used them
to provide example) but inability of a client to correctly translate
offset in case of dense packing and more than one layout per client
file. In other words, if we say that all stripe unit sizes are equal,
problem does not go away.

-Anatoly

-----Original Message-----
From: Muntz, Daniel [mailto:[email-masked]] 
Sent: Thursday, November 15, 2007 1:47 PM
To: Anatoly Pinchuk; [email-masked]
Subject: RE: [nfsv4] pNFS Dense Packing Problem

  -Dan

-----Original Message-----
From: Anatoly Pinchuk [mailto:[email-masked]] 
Sent: Thursday, November 15, 2007 12:06 PM
To: [email-masked]
Subject: RE: [nfsv4] pNFS Dense Packing Problem

 

-Anatoly

---

From: Garth Goodson <[email-masked]>
Date: Thu, 15 Nov 2007 17:25:51 -0800

> I plan to start ripping it out in a few hours unless someone can

Granted I haven't paid as much attention to the new layout structure, 
but when it was defined previously it wasn't a problem to support most 
of these options, with one exception described below.

If you wanted to use different stripe units for different sections of 
the file you would get different layouts that covered those portions. 
As long as the FHs were different you could even use DENSE and SPARSE 
layouts on different portions of the file and this may be true today as 
well.

The one exception (with the old layout scheme) is that you couldn't have 
different stripe widths for data servers covered by the same layout.  We 
felt this a reasonable compromise.  I'm guessing that the motivation to 
support this is to be able to offload more work to some servers and less 
to others which may be useful in some heterogeneous storage environments.

It seems that the new layout scheme isn't the end-all solution we were 
hoping for.  However, at some point the layout definition will need to 
stop changing -- we won't ever agree on all the changes it requires to 
support everyone's needs, and I'm afraid it just becomes more 
complicated with minimal returns...

As far as I recall, the IBM GPFS guys wanted dense layouts as that is 
how GPFS stores the data.  Using either DENSE layouts or converting 
SPARSE to DENSE on the data server requires full knowledge of the 
layout, so this shouldn't be a major stumbling block.  Maybe Marc Eshel 
can weigh in.

-Garth

---

From: Marc Eshel <[email-masked]>
Date: Fri, 16 Nov 2007 09:03:48 -0800

> > the NFSv4.1 pNFS file layout specification since as Anatoly points 
out,

> layout, so this shouldn't be a major stumbling block.  Maybe Marc Eshel 

Right now we actually use the SPARSE layout, but we can see that DENSE 
layout might be useful for some other configuration of the server that we 
might evolve to in the future. In short we would like to have it but not 
at the cost of more delays in completing this draft.
Marc.

---

From: Brent Welch <[email-masked]>
Date: Sat, 17 Nov 2007 20:43:35 -0800

FWIW, in our experience, which is all with dense packing,
is that the data servers can naturally do read-ahead w/out
knowledge of the striping.  Having large holes in the sparse
striping will likely complicate read-ahead on the data servers.
This seems like a common case that you'd like to get right,
as opposed to variable stripe units and stripe widths within
the same file.  Is the motivation for that the transition from
small files to large files?  If you are striping RAID-0 across
data servers, can't you just grow the stripe width as the files
transition from small to large?  The math on the client is trivial
with dense packing, unless of course you change the striping
in different file offsets.  Seems like an odd complication for
both sparse and dense striping.

>>>Benny Halevy said:

--
Brent Welch
Director, Software Architecture, Panasas Inc
The Leader in Parallel Storage

www.panasas.com
[email-masked]

---

From: Garth Goodson <[email-masked]>
Date: Mon, 19 Nov 2007 14:07:41 -0800

> file layout: just implement data servers to map the sparse offsets

The only downside here is that the data server must understand the full 
layout to be able to do the translation from sparse to dense.  But this 
is probably the best compromise...

-Garth