Subject: RE: [nfsv4] RE: EOF changes (truncation)

From: "Halevy, Benny" <[email-masked]>
Date: Wed, 4 Jan 2006 04:55:36 -0500

> Question: is this really a problem in the file/object-layout case?  If
> there is ambiguity the Metadata server can go query the size of the file
> by querying the data servers.  Thus, if a layoutcommit comes in
> immediately after a truncate the metadata server can go check the data
> file sizes to see what the right size should be (it can resolve the race
> by doing this); remember the size (lastbytewritten) in layoutcommit is
> merely a hint to help the metadata set the appropriate size). This also
> assumes that the truncate does not return until the data file on the
> data server has been appropriately truncated.
> 
> -Garth

Let's discuss only simple and striped files for a moment (since
writing to and truncating files in RAID layout requires synchronization
anyway).  There are two cases where clients or the MDS can go wrong
regarding the file's length.  [Let's define "storage_length" to be the
actual length of the file on storage and "client_length" what the client
thinks the length is]
Case 1: storage_length > client_length and there are non-zero data beyond
client length.  If the client (or MDS) always read from storage beyond
client_length if they need to they will see any non-zero data written there
so indeed there's a good chance there isn't any problem here, but

Case 2: storage_length < client_length and there are non-zero data cached
on the client cache beyond storage_length.  This happens after the truncate
until the client gets the cb_sizechanged callback.  In this case if a part
of a block is written beyond storage_length it might be padded with stale
non-zero data resulting in data corruption.  It might be theoretically
possible to manage the client's cache so that on cb_sizechanged the
client will clean up its cache by keeping track of the exact byte ranges
that are dirty in the cache and invalidating all the clean data beyond
the new length, but I think that there's high potential for bugs in this
area and I'm not sure it's worth the effort.

In summary, I still think that the right thing to do is to recall
layouts on truncate since: a. this is simpler and safer, and b. why
optimize for truncates?

Benny

-----Original Message-----
From: Garth Goodson [mailto:[email-masked]] 
Sent: Wednesday, January 04, 2006 1:51
To: [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nfsv4] RE: EOF changes (truncation)

[email-masked] wrote:
> Benny,
> 
> 
>>David, this approach can be implemented with pnfs CB_SIZECHANGED
>>(DESCRIPTION copied below for reference).
> 
> 
> Right, I knew that callback was there, as I'd asked for it a long time
ago.
> 
> 
>>The sequence number semantics you described below do not exist in pnfs and
>>I'm not sure if we absolutely need them.
> 
> 
> Right, HighRoad used sequence numbers, and I know we've been trying to
> keep them out of pNFS (they were initially present as layout stateids,
> but they were removed [IIRC] for simplicity).  My primary concern is
> to make sure we've gone through the design space in enough detail to
> understand the decisions we've made and their consequences.
> 
> In the situation where there's a commit that is concurrent with a
> conflicting sizechanged callback, the sequence number approach drops
> the commit on the floor if it could cause problems, whereas trying
> to solve this without sequence numbers appears to run the risk of the
> commit executing out-of-order (e.g., it's delayed in transit and arrives
> after the sizechanged callback and reply).  Sessions don't solve this
> problem, and so allowing this concurrency would appear to require a
> separate SIZECHANGED_CONFIRM operation since the callback needs to be
> responded to immediately, and that new operation does not seem like
> a good idea.
> 

Question: is this really a problem in the file/object-layout case?  If 
there is ambiguity the Metadata server can go query the size of the file 
by querying the data servers.  Thus, if a layoutcommit comes in 
immediately after a truncate the metadata server can go check the data 
file sizes to see what the right size should be (it can resolve the race 
by doing this); remember the size (lastbytewritten) in layoutcommit is 
merely a hint to help the metadata set the appropriate size). This also 
assumes that the truncate does not return until the data file on the 
data server has been appropriately truncated.

-Garth

> Related note: The first paragraph in Section 7.5.3 of the 4.1 draft says
> that pNFS relies on sessions for ordering.  I don't think that works,
> and hence I think that text needs to be removed.  My current understanding
> of sessions is that they order operations that use the same slotid within
> a session, but do not globally order operations in a session.  IMHO, it
> would be wrong to restrict all operations on any individual layout to
> a single slotid (I can already hear Tom gagging on that idea ;-) ).
> 
> This suggests that all of us need to carefully re-check the pNFS
> protocol for races based on out-of-order operation execution,
> RPCs getting reordered on the wire, etc. because sessions won't help.
> 
> Returning to the EOF behavior:
> 
> 
>>At the same time I'm not sure how
>>to resolve the race happening when the server truncates the file while
> 
> other
> 
>>clients may extend it by writing using the layout. How would the server
> 
> know
> 
>>who was the last one to set the file's length - the server or any of the
>>clients?  For example:
>>
>>1. In parallel:
>>  1.a Client A writes up to offset L
>>  1.b Server truncates the file to offset S (S < L) on behalf of client
> 
> B's
> 
>>SETATTR.
>>
>>2. In parallel:
>>  2.a Server sends CB_SIZECHANGED with size=S to client A
>>  2.b Client A sends LAYOUT_COMMIT with total_write_size=L
> 
> 
> In HighRoad, the writing client resolves the race.  If I've understood
> the scenario correctly, B's SETATTR gets ordered after A's writes
> courtesy of CB_SIZECHANGED showing up after A's writes.
> 
> 
>>The server must reject LAYOUT_COMMIT while CB_SIZECHANGED is in flight
> 
> 
> It's actually worse - the LAYOUT_COMMIT must be rejected after
> CB_SIZECHANGED is complete in order to avoid the wrong file size.
> 
> 
>>but can the client trust size=S?  Is the file size really S and not L?
> 
> 
> The client should assume that the CB_SIZECHANGED is the propagation
> of the SETATTR, and then order the client's writes as having occurred
> before vs. after arrival of that callback.  If all the client's writes
> occurred before the callback arrived, the size is S.  If the client
> has a write that extends the file to L after the arrival of the callback,
> the size is L.
> 
> 
>>With OSDs we can theoretically solve that by rotating the capability
> 
> version
> 
>>on the object atomically with truncating it in order to implicitly
>>invalidate any outstanding caps and this will force clients to refresh
> 
> their
> 
>>layouts - which we wanted to avoid with CB_SIZECHANGED :-/  Currently
>>Panasas simply recalls all outstanding layouts on truncate to make sure
> 
> the
> 
>>server is the only writer to the file.
> 
> 
> I think I'm headed here as well, but will need to think about it some
more.
> At the very least, we're going to have to write this up in Section 7.4.2
> of the 4.1 draft which currently doesn't contain anything that alerts
> implementers to the possible problem here.
> 
> In essence, because HighRoad ordered all client operations wrt server
> operations (every client operation carries a server sequence number),
> certain sequencing decisions (did the commit happen before or after
> the callback) can be made at the client with the knowledge that stale
> operations will get dropped on the floor at the server.  With pNFS not
> having done this ordering, one has to fall back to the server doing
> the sequencing.  For the truncate case, that results in a write layout
> recall - as long as the "truncate in presence of conflicting write layout
> at another client" case is rare, the simple brute force approach of
> recalling the layout may be the better way to handle this; this also.
> We should discuss this on the 4.1 draft call, and I do share your
> reluctance to bring back layout stateids.
> 
> At least in the blocks world, there's another race or races
> involving who is supposed to zero the remainder of the block that
> contains the new EOF in the extension case because one can't ask
> the storage server how long the file really is (in the blocks world,
> it doesn't know).  HighRoad uses an error/status return value to 
> tell the server that a client zeroed the remainder of the block
> containing the new EOF.  If the server doesn't get this value back
> from the client with the write layout for that block, the server has
> to do the zeroing itself, which involves recalling the layout and
> a read-modify[to zero]-write sequence.  I'm not sure what to do for
> pNFS, as this is a nasty case where the type of layout affects what
> is done in the layout-independent portion of pNFS (don't have to
> zero the rest of the block in the file or object layouts, as the
> storage server handles this).  Do we want to force a layout recall
> whenever the EOF moves?  In any case, I'll add a section on EOF
> issues to the (coming real soon now) next version of the pNFS blocks
> draft.
> 
> Thanks,
> --David
> ----------------------------------------------------
> David L. Black, Senior Technologist
> EMC Corporation, 176 South St., Hopkinton, MA  01748
> +1 (508) 293-7953             FAX: +1 (508) 293-7786
> [email-masked]        Mobile: +1 (978) 394-7754
> ----------------------------------------------------
> 
> 
>>-----Original Message-----
>>From: Halevy, Benny [mailto:[email-masked]] 
>>Sent: Friday, December 30, 2005 7:25 AM
>>To: Black, David; Halevy, Benny; [email-masked]; 
>>[email-masked]; Welch, Brent
>>Cc: [email-masked]
>>Subject: RE: EOF changes (truncation)
>>
>>David, this approach can be implemented with pnfs CB_SIZECHANGED
>>(DESCRIPTION copied below for reference). 
>>
>>   DESCRIPTION
>>
>>   The CB_SIZECHANGED operation is used to notify the client that the
>>   size pertaining to the filehandle associated with "fh", has changed.
>>   The new size is specified.  Upon reception of this notification
>>   callback, the client should update its internal size for the file.
>>   If the layout being held for the file is of the NFSv4 file layout
>>   type, then the size field within that layout should be updated (see
>>   Section 5.5).  For other layout types see Section 3.4.2 for more
>>   details.
>>
>>   If the handle specified is not one for which the client holds a
>>   layout, an NFS4ERR_BADHANDLE error is returned.
>>
>>The sequence number semantics you described below do not exist in pnfs and
>>I'm not sure if we absolutely need them. At the same time I'm not sure how
>>to resolve the race happening when the server truncates the file while
> 
> other
> 
>>clients may extend it by writing using the layout. How would the server
> 
> know
> 
>>who was the last one to set the file's length - the server or any of the
>>clients?  For example:
>>
>>1. In parallel:
>>  1.a Client A writes up to offset L
>>  1.b Server truncates the file to offset S (S < L) on behalf of client
> 
> B's
> 
>>SETATTR.
>>
>>2. In parallel:
>>  2.a Server sends CB_SIZECHANGED with size=S to client A
>>  2.b Client A sends LAYOUT_COMMIT with total_write_size=L
>>
>>The server must reject LAYOUT_COMMIT while CB_SIZECHANGED is in flight but
>>can the client trust size=S?  Is the file size really S and not L?
>>With OSDs we can theoretically solve that by rotating the capability
> 
> version
> 
>>on the object atomically with truncating it in order to implicitly
>>invalidate any outstanding caps and this will force clients to refresh
> 
> their
> 
>>layouts - which we wanted to avoid with CB_SIZECHANGED :-/  Currently
>>Panasas simply recalls all outstanding layouts on truncate to make sure
> 
> the
> 
>>server is the only writer to the file.
>>
>>If we want do better than that I'm inclined to add quiesce and resume
>>callbacks to be sent to clients with write layouts while truncating the
> 
> file
> 
>>(and possibly in other cases, e.g. for taking a snapshot).  When all
>>CB_QUIESCE calls complete successfully the server can truncate the file
> 
> and
> 
>>then send CB_SIZECHANGED+CB_RESUME callbacks meanwhile rejecting any
>>LAYOUTGET, LAYOUTCOMMIT, and LAYOUTRETURN.
>>
>>Benny
>>
>>-----Original Message-----
>>From: [email-masked] [mailto:[email-masked]] 
>>Sent: Friday, December 30, 2005 1:19 AM
>>To: [email-masked]; [email-masked]; 
>>[email-masked]
>>Cc: [email-masked]
>>Subject: EOF changes (truncation)
>>
>>Catching up on old email (December's been a very busy month)...
>>
>>
>>>>>Dean Hildebrand
>>>>
>>>>Garth Goodson
>>>
>>>Benny Halevy
>>
>>David Black
>>
>>
>>>>>2) Trond and I were talking about some odd truncate scenarios with
> 
> the
> 
>>>>>lastbytewritten field.
>>>>>s1) Client A believe the file is 20 bytes.  Client retrieves a
> 
> layout for
> 
>>>>>and writes to bytes 10-14.
>>>>>s2) Client B truncates the file to 12 bytes.
>>>>>s3) Client A performs LAYOUTCOMMIT with lastbytewritten=14 (although
> 
> it
> 
>>>>>still believes the file is 20 bytes)
>>>>>
>>>>>In this scenario, it would be best for the server to update the file
> 
> size
> 
>>>>>to 15.  If lastbytewritten was changed to local_file_size, it would
> 
> be set
> 
>>>>>to 20, which is wrong.
>>>>>
>>>>
>>>>Remember, last_byte_written is a hint for the server so that it knows
> 
> if
> 
>>>>it should put in the work to figure out the file size.  At
> 
> LAYOUTCOMMIT 
> 
>>>>time the server is free to query the data servers to find out the real
> 
> 
>>>>file size (at least for the file-based layout).  LAYOUTCOMMIT should
> 
> not
> 
>>>>be used to truncate the file (SETATTR should be used for this
> 
> purpose).
> 
>>>>So it depends on if the server received Client B's SETATTR for 12
> 
> bytes 
> 
>>>>prior to receiving the LAYOUTCOMMIT.  Also, I believe (for file-based 
>>>>layouts), the server should verify the size if it is changing the
> 
> file's
> 
>>>>size at LAYOUTCOMMIT time (it can use the current file size and the
> 
> last
> 
>>>>byte written to make this judgement).
>>>
>>>Agreed.  If the server allowed the truncate while other clients
>>>are holding layouts (and it didn't recall them), it doesn't know
>>>if the lastbytewritten refers to a write that happened before
>>>or after the truncate.  Therefore the server must query the storage
>>>servers to determine the actual file length.  That said, I'd 
>>>recommend that the MDS will recall write layouts when getting
>>>a length-changing SETATTR to force serialization across the
>>>truncate.
>>
>>One could certainly do that, but it's rather disruptive when there
>>are outstanding layouts.  I agree with Benny about the underlying
>>principle - there needs to be a synchronized view of what has 
>>happened.
>>
>>HighRoad handles this by sending a "set EOF" callback to all clients
>>that have a layout for the file and specifying that the server must
>>not handle commits that may be affected by the EOF change until all
>>the clients get the EOF change.  Some of this happens automagically
>>as the callback bumps the sequence number causing any in-flight
>>commit to be rejected with a stale sequence number.
>>
>>The advantage of this approach is that it forces clients to resolve
>>whether a write occurred before vs. after the truncate.  In the above
>>example, both 15 and 13 could be correct answers for how long the file
>>is, and client A needs to make the decision as to whether its write
>>occurred before or after B's truncate, as communicated by the EOF
>>callback.  The EOF callback forces that decision to be made, and
>>does so without any layout recalls.  If A) and B) are communicating
>>out of band, pNFS as currently specified can produce 
>>incorrect behavior:
>>
>>s1) Client A believe the file is 20 bytes.  Client retrieves a layout
>>	for and writes to bytes 10-14.
>>New: s1.a) Client A tells client B that it has written to the file
>>s2) Client B truncates the file to 12 bytes.
>>
>>In this situation, the only right answer for the file length is 13,
>>and B's truncate must be passed to A to make this happen.
>>
>>The disadvantage of this approach is that the EOF handling code can
>>be somewhat subtle (e.g., in the above example in the block world,
>>client A is responsible for zeroing out the block beyond the new
>>EOF because it holds the only write extent for that block).
>>
>>HighRoad is specified in terms of clients passing the new EOF on a
>>commit (because the client always receives notification of truncates),
>>but the server behavior is equivalent to the pNFS last_byte_written
>>approach because the new EOF can only extend the file - if a client
>>wants to truncate, it has to issue a SETATTR to do so.
>>
>>There's a new version of the pNFS block draft coming shortly - I'll
>>make sure this issue is noted in it.  I think we have some more
>>design work to do.
>>
>>Thanks,
>>--David
>>----------------------------------------------------
>>David L. Black, Senior Technologist
>>EMC Corporation, 176 South St., Hopkinton, MA  01748
>>+1 (508) 293-7953             FAX: +1 (508) 293-7786
>>[email-masked]        Mobile: +1 (978) 394-7754
>>----------------------------------------------------
>>
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/nfsv4

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4

---

From: Garth Goodson <[email-masked]>
Date: Wed, 04 Jan 2006 09:36:03 -0800

> layouts on truncate since: a. this is simpler and safer, and b. why

I'm just trying to point out that implementations can handle things in 
different ways.  In fact some implementations may always keep attributes 
synchronized (there are many ways to do this).  I don't want to dictate 
that layouts MUST be recalled on a truncate (especially for file-based 
layouts; block-layouts may incur this restriction as the metadata server 
has less knowledge about the actual file state).  I would rather we 
dictate what the semantics should be and give an example of how to 
achieve them, rather than dictating the mechanism.

Changing topics: Another note on the usefulness of the cb_sizechanged 
callback for file-based layouts... The cb_sizechanged callback is also 
useful for clients who only hold a read-layout in the case when a read 
is issued to a file stripe whose size on the data server is less than 
the read offset, but whose file size is greater than the amount being 
read (so reading a hole in a file that appears at the end of a file 
stripe on the data server, but not after the end of the file).  In this 
case the client needs to know to return zeros, even though it got an EOF 
back from the data server read.  This is described in more detail in the 
spec.

-Garth