Subject: RE: [nfsv4] Issue 21: pnfs EOF handling

From: [email-masked]
Date: Mon, 10 Jul 2006 16:46:28 -0400

Hi all--
Garth has proposed that this topic be discussed in Montreal
tomorrow... feel free to use the text below as a strawman.

Here is an update to my proposed text on the issue.  I've
been monitoring the list discussion, and I think we have
rough consensus on the following principles:

1. Object and file layout semantics require that clients
are updated both when a file is truncated and when a file
is extended.

2. Block layout semantics are different for the truncate
case and the extend case.

3. CB_SIZECHANGED has NO recall semantics.

4. For block layout, the CB_SIZECHANGED has side-effects
on layouts held by the client (layouts beyond the new eof
must be re-committed to the server).

5. For block layout servers, we will synchronize the client
state by pending the SETATTR, sending the CB_SIZECHANGED,
waiting for the replies, then completing the SETATTR.
DELAY will not be returned to the client requesting the
SETATTR once the CB_SIZECHANGED callbacks have been sent.

6. Text in the generic draft will not mandate any sort
of processing that is specific to a single layout-type.

Looking at these principles, I think its clear that each
layout type will handle size changes quite differently. For
this reason, I found it hard to say anything too specific
in the generic draft about how and when CB_SIZECHANGED
should be issued.  Obviously, the layout-specific specifications
should be explicit about the requirements for sending
and waiting for these callbacks.

Anyway, with all of the above in mind, here is what I have
come up with:

In section 14.4.2, replace the following paragraph:

Since client layout holders may be unaware of changes made to the
file's size, through LAYOUTCOMMIT or SETATTR, by other clients, an
additional callback/notification has been added for pNFS.
CB_SIZECHANGED is a notification that the metadata server sends to
layout holders to notify them of a change in file size.  This is
preferred over issuing CB_LAYOUTRECALL to each of the layout holders.

with the following text:

Since changes made to the file's size may affect the semantics of
layouts held by other clients, an additional callback/notification
has been added for pNFS.  CB_SIZECHANGED is a notification that the
metadata server sends to layout holders to notify them of a change
in file size.  The callback contains a field that allows the server
to indicate the source of the change in size:  either an explicit
request from a client via SETATTR, or an implicit extension of the
file size via a WRITE or a LAYOUTCOMMIT.  Depending on the
requirements of the storage protocol, the server MAY issue
CB_SIZECHANGED callbacks to clients which hold layouts on a file
whenever that file's size is changed.  Again depending on the
requirements of the storage protocol the server MAY issue
CB_LAYOUTRECALL callbacks for portions of layouts which overlap or
lie beyond the new end of file.  If both are issued, the
CB_LAYOUTRECALL callback SHOULD follow the CB_SIZECHANGED callback
in the same compound callback operation.  Each layout-specific
specification will enumerate the precise requirements for when and
how CB_SIZECHANGED and CB_LAYOUTRECALL callbacks MUST be issued
in response to each change in file size.

And here is the new section 28 with my additions marked:

28.  CB_SIZECHANGED

28.1.  SYNOPSIS

   fh, size -> -

28.2.  ARGUMENT

*   enum sizechanged_reason4 {
*       SIZECHANGED_SETATTR = 1,
*       SIZECHANGED_EXTEND = 2
*   };

   struct CB_SIZECHANGEDargs {
       nfs_fh4               fh;
*       sizechanged_reason4   reason;
       length4               size;
   };

28.3.  RESULT

   struct CB_SIZECHANGEDres {
       nfsstat4        status;
   };

28.4.  DESCRIPTION

   The CB_SIZECHANGED operation is used to notify the client that the
   size pertaining to the filehandle associated with "fh", has changed.
*   The new size is specified.  The "reason" field indicates whether
*   the change in size was the result of an explicit change via a
*   SETATTR request, or an implicit extension of the file due to a
*   WRITE or LAYOUTCOMMIT. Upon reception of this notification
   callback, the client SHOULD update its internal size for the file.
*   Depending on the storage protocol used, the client may also need to
*   update the states of layouts it holds for the file which span the
*   new end of file byte.  For example, if the layout being held for the
*   file is of the NFSv4 file layout type, then the size field within
*   that layout should be updated (see Section 16.5).  For other layout
*   types, the layout-specific specification will provide details of the
*   updates that are required.

   If the handle specified is not one for which the client holds a
   layout, an NFS4ERR_BADHANDLE error is returned.

28.5.  IMPLEMENTATION

28.6.  ERRORS

   NFS4ERR_BADHANDLE

> -----Original Message-----
> From: Benny Halevy [mailto:[email-masked]] 
> Sent: Monday, July 10, 2006 2:05 PM
> To: Garth Goodson
> Cc: Black, David; [email-masked]
> Subject: Re: [nfsv4] Issue 21: pnfs EOF handling
> 
> Garth Goodson wrote:
> 
> >
> >
> > Benny Halevy wrote:
> >
> >> [email-masked] wrote:
> >>
> >>> The simplest fix may be to only issue CB_SIZECHANGED for 
> an explicit
> >>> SETATTR,
> >>> and never for any implicit EOF change (write or commit).  A more
> >>> comprehensive
> >>> approach would be to add a "why" flag to CB_SIZECHANGED 
> to distinguish
> >>> implicit extension from explicit length set, although I can't 
> >>> immediately
> >>> figure out why a pNFS client needs to see an implicit EOF 
> change for 
> >>> file
> >>> extension. 
> >>
> >>
> >>
> >> [snip]
> >>
> >> David, In the objects case, when the file is striped over multiple 
> >> objects and the
> >> client gets a read beyond its notion of the EOF it has to go read 
> >> from storage
> >> to determine the current file length or issue a GETATTR to the MDS.
> >> If the file is holey and was sparsely written by another client 
> >> beyond this client's
> >> notion of the EOF it is possible that the client will get a short 
> >> read from one
> >> of the objects but there's more data in higher offsets on 
> other objects.
> >> In order to make this data visible the client needs to 
> know the new EOF.
> >>
> >> Benny
> >>
> >
> > Same for files.  Alternatively, extending writes could extend the 
> > files on the data servers.
> 
> This has a similar race between writers since the SETATTR operation 
> would need an "extend only"
> flag to make sure nobody else's data is being truncated.
> 
> >
> > -Garth
> >
> >>
> >> _______________________________________________
> >> nfsv4 mailing list
> >> [email-masked]
> >> https://www1.ietf.org/mailman/listinfo/nfsv4
> >
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4
> 

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Trond Myklebust <[email-masked]>
Date: Mon, 10 Jul 2006 22:39:28 -0400

> are updated both when a file is truncated and when a file

Why? Standard NFS close-to-open semantics do not require this.

Trond

---

From: Garth Goodson <[email-masked]>
Date: Mon, 10 Jul 2006 20:43:40 -0700

> Why? Standard NFS close-to-open semantics do not require this.

True.  However, this is where pNFS tries to give back some of the 
looseness is has created through the use of LAYOUTCOMMIT.  The issue is 
that even though a write might be COMMITed (note, not LAYOUTCOMMITed), a 
reader may see an EOF if it reads a hole in a sparse file.  While this 
technically doesn't break open-close semantics, the EOF callback can 
help bridge this gap and provide the client more information about the 
actual length of the file, so it will know it is just reading holes.

This mechanism is just trying to provide a more similar type of behavior 
compared with a single NFS server implementation.  (Aside: to compound 
this issue, no post-op attrs can be usefully obtained from the DSs, so 
it becomes harder for a client to know when to query the MDS attrs).

-Garth

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 11 Jul 2006 01:30:36 -0400

> it becomes harder for a client to know when to query the MDS attrs).

Why would this be better than the lazy alternative (i.e. recall the
layout when someone else requests a GETATTR call)?

Cheers,
  Trond

---

From: Benny Halevy <[email-masked]>
Date: Tue, 11 Jul 2006 06:37:55 -0400

>layout when someone else requests a GETATTR call)?
I think there's no real need to recall the layout for files and objects 
since only data and attributes
were changed by the writer. Wouldn't using CB_GETATTR be more efficient 
in this model?

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 11 Jul 2006 07:57:20 -0400

> were changed by the writer. Wouldn't using CB_GETATTR be more efficient 

I suppose so. At least if the other client is just reading the file.

Note that the client is not guaranteed to have a delegation for the
file, so it would be a change in behaviour for CB_GETATTR.

Also note that the concept of uncertainty in the file length is hardly
new: we've had that ever since we introduced write caching. By that
precedent, it would be perfectly justifiable to allow the reader to EOF
when it hits the old end-of-file. Why does pNFS need better caching
semantics than NFS here?

Cheers,
  Trond

---

From: Garth Goodson <[email-masked]>
Date: Tue, 11 Jul 2006 05:33:29 -0700

> when it hits the old end-of-file. Why does pNFS need better caching
While the semantics may seem better than NFS I don't think they really 
are.  Basically you are getting what pretty much any NFS server gives 
you --- the ability for a client that issued a read to see the latest 
data comitted at the NFS server.  I agree this is better than 
open-close, but it is what you get today.  Also, since the EOF callback 
was proposed for blocks, we thought we could make use of it.

You are correct that it would be justifiable to return EOF to the 
reader.  But as I mentioned earlier, it is slightly harder for the 
reader to obtain the new EOF (it can't piggyback getattr calls on the 
read).

I think we are arguing a minor point here.  The EOF callback is an 
optimization.

-Garth

---

From: Trond Myklebust <[email-masked]>
Date: Tue, 11 Jul 2006 09:12:04 -0400

> was proposed for blocks, we thought we could make use of it.

What you get today is that the client read will see whatever the server
sees. That is pretty much what you get without a callback at all.

Why does blocks need it?

> I think we are arguing a minor point here.  The EOF callback is an 

My question is whether or not we need this type of optimisation at all.
If it doesn't add anything useful to the protocol, I'd vote to kill it.

Cheers,
  Trond