Subject: [nfsv4] LAYOUTGET minlength

From: Benny Halevy <[email-masked]>
Date: Wed, 14 Nov 2007 15:49:05 +0200

Following a discussion we had in the last Bakeathon
I propose the following wording changes:

1. Allow zero minlength.
The motivation is to send an opportunistic LAYOUTGET for
iomode=open_mode and offset=length=minlength=0 along with OPEN
so that the server can return an initial layout starting from offset 0
for which the length doesn't matter, as the client has no bytes to
read or write at this point.

diff -Npu a/nfsv41_middle_pnfs.xml b/nfsv41_middle_pnfs.xml
--- a/nfsv41_middle_pnfs.xml
+++ b/nfsv41_middle_pnfs.xml
@@ -454,9 +454,8 @@
   the requested byte range.  A field within the LAYOUTGET request,
   loga_minlength, specifies the minimum overlap that MUST exist
   between the requested layout and the layout returned by the metadata
-  server. The loga_minlength field should be at least one.  As needed a
-  client may make multiple LAYOUTGET requests; these will result in
-  multiple overlapping, non-conflicting layouts.
+  server.  As needed, a client may make multiple LAYOUTGET requests;
+  these will result in multiple overlapping, non-conflicting layouts.
 </t>
 <t>
   There is no required ordering between getting a layout and performing

2. clarify how loga_minlength relates to the requested offset
There was agreement among the bakeathon participants that the minlength
bytes that the client wants should start from the requested offset rather
than just any minlength bytes in the requested byte-range.

diff -Npu a/nfsv41_middle_op_layoutget.xml b/nfsv41_middle_op_layoutget.xml
--- a/nfsv41_middle_op_layoutget.xml
+++ b/nfsv41_middle_op_layoutget.xml
@@ -69,7 +69,8 @@
       the loga_iomode.  The client must be prepared to get a layout
       that does not align exactly with its request. There MUST be at
       least an overlap of loga_minlength between the layout returned
-      by the server and the client's request, or the server SHOULD
+      by the server and the client's request, starting from loga_offset
+      to loga_offset + loga_minlength, or the server SHOULD
       reject the request.  See <xref target="obtaining_layout" /> for
       more details.
     </t>

Benny

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 14 Nov 2007 09:29:46 -0500

I guess I don't have any objection to allowing this to be zero
but I think it is a mistake on the part of client to send this 
value as zero or anything below around 64K.

Although I'm not asking for the spec to say this, my view is
that if you send very low values, then you "SHOULD have your 
head examined."

Accessing the data server directly is an optmization and the
layout lengths you get should not make it a pessimization.
If the layout lengths are so short that it is, you should read
and write to the MDS.  A value of zero, which suggests that
you want any length for communication with the DS, no matter
how small, in preference to communication with the MDS, suggests
a very different view of the matter.  We can't legislate 
performance characteristics (or at least I'd prefer not to have
to draft the requisite legislation), but I would hope that
clients not assume the sort of very slow MDS that would justify
this strategy and that servers would not drive them to it.

"for which the length doesn't matter, as the client has no bytes 
to read or write at this point." seems to ignore important issues.
Despite the fact that no read or write is physically pending,
clients will have buffer sizes and other existing performance 
characteristics that give some idea of where the cross-over 
point for MDS-vs-DS access is.  Given that, I would say that
there would be some non-zero length where the client would in
effect say "If you can't give me at least N bytes, then don't
bother".  Saying that one byte is OK whether by sending one or
zero as the minlength would seem to call for the aformentioned
mental status examination.

In the implementations I would be doing, even if the client asked
for zero, he would get a considerable length, but he would also
get that if he asked for one byte. 

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Wednesday, November 14, 2007 8:49 AM
To: NFSv4
Subject: [nfsv4] LAYOUTGET minlength

Following a discussion we had in the last Bakeathon I propose the
following wording changes:

1. Allow zero minlength.
The motivation is to send an opportunistic LAYOUTGET for
iomode=open_mode and offset=length=minlength=0 along with OPEN so that
the server can return an initial layout starting from offset 0 for which
the length doesn't matter, as the client has no bytes to read or write
at this point.

2. clarify how loga_minlength relates to the requested offset There was
agreement among the bakeathon participants that the minlength bytes that
the client wants should start from the requested offset rather than just
any minlength bytes in the requested byte-range.

diff -Npu a/nfsv41_middle_op_layoutget.xml
b/nfsv41_middle_op_layoutget.xml

Benny

---

From: Marc Eshel <[email-masked]>
Date: Wed, 14 Nov 2007 07:10:52 -0800

I think this a fine idea. It is just a way for the client to tell the 
server to deiced how much to return. If it is a read for a small file and 
the server prefers that the client don't use layout it can decline the 
request for a layout, but I think that in general it is better to use 
layout for every file so we get load balancing. If you don't use layout 
for small files and most of the files are small you are going to overload 
your MDS.
Marc.

---

From: "Noveck, Dave" <[email-masked]>
Date: Wed, 14 Nov 2007 11:07:31 -0500

> It is just a way for the client to tell the server to deiced how 
> much to return. 

Exactly.  It's saying whatever the server decides is OK with you.
I'd prefer not to do that.

> server prefers that the client don't use layout it can decline 
> the request for a layout

Doesn't the client have any preferences here?

> If you don't use layout for small files and most of the files 
> are small you are going to overload your MDS.

I think you need to distinguish small files and small layouts.  If 
a file is 50 bytes and I get a 64K layout, then I'm cool sending a
50-byte READ to the DS.  If the file is 64K and I get a 50 byte 
layout, the DS is not going to be my choice.  If the file and 
the layout are both 50 bytes, I'm not sure.  I'd probably decide
to use a different server.

-----Original Message-----
From: Marc Eshel [mailto:[email-masked]] 
Sent: Wednesday, November 14, 2007 10:11 AM
To: Noveck, Dave
Cc: Benny Halevy; NFSv4
Subject: RE: [nfsv4] LAYOUTGET minlength

I think this a fine idea. It is just a way for the client to tell the
server to deiced how much to return. If it is a read for a small file
and the server prefers that the client don't use layout it can decline
the request for a layout, but I think that in general it is better to
use layout for every file so we get load balancing. If you don't use
layout for small files and most of the files are small you are going to
overload your MDS.
Marc.

"Noveck, Dave" <[email-masked]> wrote on 11/14/2007 06:29:46 AM:

> I guess I don't have any objection to allowing this to be zero but I 
> think it is a mistake on the part of client to send this value as zero

> or anything below around 64K.
> 
> Although I'm not asking for the spec to say this, my view is that if 
> you send very low values, then you "SHOULD have your head examined."
> 
> Accessing the data server directly is an optmization and the layout 
> lengths you get should not make it a pessimization.
> If the layout lengths are so short that it is, you should read and 
> write to the MDS.  A value of zero, which suggests that you want any 
> length for communication with the DS, no matter how small, in 
> preference to communication with the MDS, suggests a very different 
> view of the matter.  We can't legislate performance characteristics 
> (or at least I'd prefer not to have to draft the requisite 
> legislation), but I would hope that clients not assume the sort of 
> very slow MDS that would justify this strategy and that servers would 
> not drive them to it.
> 
> "for which the length doesn't matter, as the client has no bytes to 
> read or write at this point." seems to ignore important issues.
> Despite the fact that no read or write is physically pending, clients 
> will have buffer sizes and other existing performance characteristics 
> that give some idea of where the cross-over point for MDS-vs-DS access

> is.  Given that, I would say that there would be some non-zero length 
> where the client would in effect say "If you can't give me at least N 
> bytes, then don't bother".  Saying that one byte is OK whether by 
> sending one or zero as the minlength would seem to call for the 
> aformentioned mental status examination.
> 
> In the implementations I would be doing, even if the client asked for 
> zero, he would get a considerable length, but he would also get that 
> if he asked for one byte.
> 

> the server can return an initial layout starting from offset 0 for 
> which the length doesn't matter, as the client has no bytes to read or

> write at this point.
> -  server. The loga_minlength field should be at least one.  As needed

> a
> 2. clarify how loga_minlength relates to the requested offset There 
> was agreement among the bakeathon participants that the minlength 
> bytes that the client wants should start from the requested offset 
> rather than just any minlength bytes in the requested byte-range.
>

---

From: Benny Halevy <[email-masked]>
Date: Thu, 15 Nov 2007 14:39:19 +0200

> Doesn't the client have any preferences here?

The idea of length=0,minlength=0 is that the client does not need
the layout right now.  Without having that, the server may provisionally
allocate storage space for no reason as the application may not
write at all to the file or write to some byte range not covered
by the initial guess.  With length=0,minlength=0 you leave that
up to the server which may return, for example in our case, the
layout with capabilities that allow you only to do getattrs to
the objects so you can make as much preparations, e.g. connect
with the respective OSDs, get each object's attributes, in advance,
without doing any I/O.  So the layout will describe the file
but will not allow any I/O.  How about allowing minlength == 0 only
for length == 0?

> to use a different server.

If you care, set minlength != 0, if not, then don't and let the server
make the choice. minlength==0 helps with allowing the server to return
a layout for no bytes and avoiding any unneeded provisional allocations.

Benny

---

From: Mike Eisler <[email-masked]>
Date: Thu, 15 Nov 2007 05:25:40 -0800 (PST)

> read or write at this point.

Why doesn't minlength=length=1 work?

Here's my preference: delete minlength. The text about minlength
and overlap never made any sense.

> rather

---

From: Mike Eisler <[email-masked]>
Date: Thu, 15 Nov 2007 05:40:38 -0800 (PST)

> value as zero or anything below around 64K.

What if I have 100 million thumbnail images that are under 
64K?

Distributing lots of little files is what pNFS is good at, or
at least should be good at. The post-Austin rework of the
files-based layout and deviceID->device address stuff was done
in part to preserve that feature.

---

From: Benny Halevy <[email-masked]>
Date: Thu, 15 Nov 2007 15:48:24 +0200

> Why doesn't minlength=length=1 work?

because for writes, you don't want the server to provisionally allocate
anything at this point.

> and overlap never made any sense.

I'm OK with that.

But I'd still prefer to require that the returned layout MUST cover
the requested offset, i.e.:
lo_offset <= loga_offset &&
lo_offset + lo_length >= loga_offset + ((loga_length != 0) ? 1 : 0);

Benny

---

From: "Noveck, Dave" <[email-masked]>
Date: Thu, 15 Nov 2007 09:16:50 -0500

> What if I have 100 million thumbnail images that are under 64K?

Then fine.  What's the problem?  The layouts tell you where (on
what machine) the first 64K of each of those would go, you read
them, with appropriate parallelism and everything is OK. 

-----Original Message-----
From: Mike Eisler [mailto:[email-masked]] 
Sent: Thursday, November 15, 2007 8:41 AM
To: Noveck, Dave; Benny Halevy; NFSv4
Subject: RE: [nfsv4] LAYOUTGET minlength

> or anything below around 64K.

What if I have 100 million thumbnail images that are under 64K?

Distributing lots of little files is what pNFS is good at, or at least
should be good at. The post-Austin rework of the files-based layout and
deviceID->device address stuff was done in part to preserve that
feature.

---

From: Spencer Shepler <[email-masked]>
Date: Thu, 15 Nov 2007 13:47:00 -0600

> lo_offset + lo_length >= loga_offset + ((loga_length != 0) ? 1 : 0);

Did you intend:

	lo_offset <= loga_offset &&
	lo_offset + lo_length >= loga_offset + (loga_length != 0) ?  
loga_length : 1

In other words the returned layout's length must be at least
equal to the length originally requested?

I am of the opinion that the server should be able to
return less that what was requested and this behavior
would be useful.

Spencer

---

From: Benny Halevy <[email-masked]>
Date: Fri, 16 Nov 2007 08:51:38 +0200

> equal to the length originally requested?

No, that's not always possible for the server.
The requirement above is just the minimum the server must return
(in the success case).  I should probably have stated that as
"The returned layout MUST cover the requested offset and at least
one more byte if loga_length is greater than zero"

> return less that what was requested and this behavior

Ditto.  To make forward progress the client must be prepared to get less
than what it requested.  My rational for:
lo_offset + lo_length >= loga_offset + ((loga_length != 0) ? 1 : 0);
if if the client indicated a loga_length != 0 then the server should
return a layout covering at least 1 byte after loga_offset, to allow
forward progress (of course 1 byte is not very practical in most case
and I count on servers to do a better job if they want to survive :)
otherwise, if loga_length was zero, the server MAY return a layout
describing the file only at offset == loga_offset.

---

From: "Noveck, Dave" <[email-masked]>
Date: Fri, 16 Nov 2007 10:24:44 -0500

> than what it requested. 

To make forward progress the client must be prepared to get zero,
because 
he is not guaranteed to get any particular amount even if that amount is
one
byte.  The client must be prepared to read and write through the MDS.
Some of the discussion here seems to assume that a layout and IO to
the DS is a sine qua non when you have pNFS active, and I don't think
that is right.

If the client gets one byte and does not take the hint and use the MDS,
then it is not going to survive.  I think it makes sense for the client
to be able to tell the server in advance what his useful minimum will
be, so if the server can't provide it, the client doesn't have a useless
layout to return.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Friday, November 16, 2007 1:52 AM
To: Spencer Shepler
Cc: [email-masked]; NFSv4
Subject: Re: [nfsv4] LAYOUTGET minlength

On Nov. 15, 2007, 21:47 +0200, Spencer Shepler <[email-masked]>
wrote:
>>>> so that the server can return an initial layout starting from
offset
>>>> 0