Subject: Re: [nfsv4] question about pNFS.

From: Benny Halevy <[email-masked]>
Date: Sun, 29 Jun 2008 09:38:33 +0300

Yi,

The server strategy for granting overlapping write layouts
and preventing corruption due to concurrent writing of the
file depends on the layout type and other parameters (e.g.
objects RAID level for pnfs-obj).

For example with the files layout or with a striped (RAID-0)
file over the objects layout, if the clients write *only*
the application buffers, i.e. they do not read around that
to pad and align to page/buffer size, then the client should
not step on each other's data, since writing to any data server
in this case is no different than writing to the metadata
server.

Other cases require serialization of writes that is out of
scope for the client.  For example, with objects RAID, the
whole stripe needs to be locked while writing to it,
therefore the server will not grant more than one outstanding
layout for any specific stripe.

Another example could be the block layout, for which the server
provisionally allocates blocks for newly written data.
In this case, false sharing may result in data corruption if
two clients are given concurrently write layouts overlapping
a block, if the server is unable to perform LAYOUTCOMMIT
for extents that are not block aligned.

Benny

On Jun. 26, 2008, 15:02 +0300, "易乐天" <[email-masked]> wrote:
>    Hmm..   i do not mind adding it to mail list, but i think some
> thing(the second method) i considered is wrong...
>  
>    In the situatition i refered, two client just write concurrently a
> shared file, and this two client write the same range of file(may be the
> same data) by turns with no lock and data reservation, the server have
> two method to grant layout. the first is the same as method described in
> "draft of pNFS", and the second method means server can grant the layout
> refer the same range(may be a block or a object of a file) to two
> clients, and this may lead to no written of bytes of one client   
> 
>  
> 2008/6/24 Benny Halevy <[email-masked] <mailto:[email-masked]>>:
> 
>     Hi Yi,
> 
>     I'm not sure I understand exactly the workload you described
>     so I appreciate if you can give more concrete example
>     describing when the different clients are writing
>     and to which regions of the file.
> 
>     Also, this question is really appropriate for the nfsv4
>     mailing list, so do you mind if I add it to the Cc on
>     my reply?
> 
>     Benny
> 
>     On Jun. 24, 2008, 9:39 +0300, "易乐天" <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     > hi Benny
>     >       I've read pNFS draft these days, a problem about data caching
>     > still puzzle me.
>     >       Your explaination about range lock may be good enough to
>     solve to
>     > some sophisticated application with range lock request to parallel
>     > accessing the same file(some process to access the same file).
>     However,
>     > there may be some infantile application with no range lock. In this
>     > scenes, as you said, two client just write concurrently a shared file,
>     > and this two client write file by turns. Of course, here, both client
>     > restrict nerther the data reservation nor range lock. In this
>     > situation, because of the some other reasons, i think the content
>     of the
>     > file may be "inconsistency" of both process after closing.
>     >       what puzzled me is, in the processing of above, which
>     > method should MDS apply:
>     >      1. recall and grant layout of this file to clients
>     frequently. And
>     > this may be benefit from "layout prefecth". for example, client A
>     access
>     > the first 1G range of FILE, and B access the second 1G range.The
>     > server can grant layout of both range to these client and
>     eliminate the
>     > reference to server from client.
>     >      2. no layout caching and data caching in any clients. this
>     may be a
>     > conservative method.
>     >
>     >
>      
>     > 2008/6/2, Benny Halevy <[email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>>:
>     >
>     >     Basically, the application lock requests turn into
>     >     NFSv4 byte range locks that are sent to the metadata server.
>     >     Note that NFSv4 has fine grain locks, not only coarse grain
>     >     share reservations.
>     >
>     >     With clustered back-end file systems like Panasas, GPFS, or GFS2
>     >     these might turn into DLM calls in the back-end cluster.
>     >
>     >     If the clients just read and write concurrently from a shared
>     >     file it is up to the back-end file system to coordinate the
>     >     access.  This is done by recalling and granting layouts
>     >     appropriately when needed, depending on the file's striping
>     >     pattern and on the back-end file system's architecture.
>     >     So essentially this is no less efficient than any other
>     >     clustered file system.  Files striped with RAID-0 need
>     >     no synchronization between pnfs clients, however, for
>     >     back-end filesystems like GPFS or GFS2, they need to utilize
>     >     the DLM "behind the scenes" to coordinate their internal
>     >     access to shared storage.  Panasas, however, using object-based
>     >     storage in a partitioned storage architecture need no global
>     >     synchronization between the OSDs.  If the the file is striped
>     >     in RAID-5, like we typically do, we do coordinate access
>     >     by recalling layouts so that no two pnfs clients could
>     >     write to the same stripe at the same time.  If each client
>     >     writes a big enough region this is very effective and
>     >     we can reach multi-gigabyte per second throughput even
>     >     with RAID-5.  For smaller writes, and if the clients'
>     >     regions do not overlap we'd typically recommend
>     >     striping with RAID-10 or (RAID-0) and get even better
>     >     performance (in the expense of space (RAID-10) or reliability
>     >     (RAID-0)).
>     >
>     >     Benny
>     >
>     >     On Jun. 02, 2008, 10:31 +0300, "易乐天" <[email-masked]
>     <mailto:[email-masked]>
>      
>     >     <mailto:[email-masked] <mailto:[email-masked]>>>
>     wrote:
>     >     > hi  benny
>     >     >     According to the goals of pNFS, client should not only
>     access
>     >     the shared
>     >     > storage, but also can support high end scientifical computing.
>     >     However,
>     >     > because of the limit of naive lock of NFS, for example LEASE
>     (it is
>     >     > coarse-granularity-lock which may only be used to protect entire
>     >     file), it
>     >     > can not support the concurrence accessing which exists in
>     many HPC's
>     >     > application.And in some cluster file system such as lustre, GFS,
>     >     > GPFS, they use Distributed Lock Manager to achieve it.
>     >     >     So, I am afraid that pNFS can not do the same work
>     effeciently
>     >     as above
>     >     > file system if no DLM be added in it. And how do you think
>     of it?
>     >     >
>     >     >
>     >     > 2008/5/28 易乐天 <[email-masked]
>     <mailto:[email-masked]>
>      
>     >     <mailto:[email-masked] <mailto:[email-masked]>>>:
>     >     >
>     >     >> hi Benny
>     >     >>   Thank you reply about this questions and your reply helps
>     me to
>     >     >> understand pNFS more deeply.
>     >     >>
>     >     >> 2008/5/27 Benny Halevy <[email-masked]
>     <mailto:[email-masked]>
>      
>     >     <mailto:[email-masked] <mailto:[email-masked]>>>:
>     >     >>
>     >     >>   易乐天 wrote:
>     >     >>>> hello halevy.
>     >     >>>>    I am interesting in pNFS, but i have a question about it.
>     >     >>>>   Except the pluggable-layout-driver that can support
>     "FILE, BLOCK,
>     >     >>>> OBJECT" model, i can not find out the other advantages or
>     >     attracting
>     >     >>>> aspects of pNFS when compare it with other cluster file
>     system
>     >     such as
>     >     >>>> lustre, storage tank, panFS. With these fs, users can
>     select and
>     >     >>>> obtain all model(FILE, BLOCK, OBJECT) according to their
>     hardware
>     >     >>>> environment.
>     >     >>>>   Can you explain it to me : what characters pNFS owed while
>     >     comparing
>     >     >>>> it with these cluster file system?
>     >     >>>>   thank you!
>     >     >>>>
>     >     >>>> --
>     >     >>>> Best regards.
>     >     >>>>
>     >     >>>> Yi LeTian
>     >     >>>>
>     >     >>>> School of Computer Science, Changsha, Hunan, China
>     >     >>>>
>     >     >>>> Email: [email-masked]
>     <mailto:[email-masked]> <mailto:[email-masked]
>     <mailto:[email-masked]>>
>      
>     >     <mailto:[email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>>
>     >     >>> The main advantage of pNFS comparing to any other proprietary
>     >     solution
>     >     >>> is that it
>     >     >>> is an IETF standard that's layered on top of other
>     standard storage
>     >     >>> protocols.
>     >     >>> For customers this means that they can potentially deploy
>     >     best-of-breed
>     >     >>> solution
>     >     >>> choosing from multiple vendors.
>     >     >>>
>     >     >>> From the technical perspective only, we tried to combine
>     features
>     >     >>> already present
>     >     >>> in several existing and similar architectures like Panasas,
>     >     Lustre, EMC
>     >     >>> high road, etc.
>     >     >>> and not necessarily improve on them.  One thing that is in the
>     >     standard
>     >     >>> and
>     >     >>> may be missing from some existing clustered file system is the
>     >     propagation
>     >     >>> of device information that allows the client to dynamically
>     >     discover and
>     >     >>> mount the storage devices without having to configure them on
>     >     each and
>     >     >>> ever client.
>     >     >>>
>     >     >>> Also, using NFSv4.1 infrastructure, NFSv4.1 sessions provide
>     >     >>> "exactly-once-semantics"
>     >     >>> and recovery features that may be missing from some existing
>     >     file systems.
>     >     >>>
>     >     >>> I hope this answers your question.
>     >     >>>
>     >     >>> Benny
>     >     >>>
>     >     >>>
>     >     >>>
>     >     >>
>     >     >> --
>     >     >> Best regards.
>     >     >>
>     >     >> Yi LeTian
>     >     >>
>     >     >> School of Computer Science,  Changsha, Hunan, China
>     >     >>
>     >     >>
>     >     >> Email: [email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>
>     >     >>
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>     >
>      
>     > --
>     > Best regards.
>     >
>     > Yi LeTian
>     >
>     > School of Computer Science, National University of Defense Technology,
>     > Changsha, Hunan, China
>     >
>     > Tel: 13467509790
>      
>     > Email: [email-masked] <mailto:[email-masked]>
>     <mailto:[email-masked] <mailto:[email-masked]>>
> 
>      
> 
> 
> 
> 
> -- 
> Best regards.
> 
> Yi LeTian
> 
> School of Computer Science, National University of Defense Technology,
> Changsha, Hunan, China
> 
> Tel: 13467509790
> Email: [email-masked] <mailto:[email-masked]>
_______________________________________________
nfsv4 mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/nfsv4

---

From: "易乐天" <[email-masked]>
Date: Sun, 29 Jun 2008 22:32:55 +0800

> layout for any specific stripe.

  as to the lock of whole stripe, do you means the RAID-5,6...(all RAID with
parity, and client must cach data to compute parity), but RAID-0,1... ? i do
not think RAID-0,1 need to lock whole stripe..

> for extents that are not block aligned.

  while doing "pre-allocate"(do not only one block be allocated), i think if
the write layouts overlapping a block is granted to two clients, and the
server can perform LAYOUTCOMMIT, the data may be corruption. for example, A
CLIENT allocate new blocks (file logic address 1M-2M), and B CLIENT get the
layout of 1M-2M (of course, both clients doing "write only" or "wirte_read"
file), and the A CLIENT commit the file of size 1.5M, the B CLIENT will
write the range of 1.5M-2M range of the file but at this time these block
may be allocated to other files.
so i think the layout should be granted to one "writing" client at any time.

---

From: "易乐天" <[email-masked]>
Date: Sun, 29 Jun 2008 22:40:37 +0800

benny
"so i think the layout should be granted to one "writing" client at any
time." in the last mail, i means the layout of the same range of file ( may
be  refer to the same block or the same object )

2008/6/29, 易乐天 <[email-masked]>:
>   as to the lock of whole stripe, do you means the RAID-5,6...(all RAID
> with parity, and client must cach data to compute parity), but RAID-0,1... ?
> i do not think RAID-0,1 need to lock whole stripe..
>
>
>
>
>   while doing "pre-allocate"(do not only one block be allocated), i think
> if the write layouts overlapping a block is granted to two clients, and the

-- 
Best regards.

Yi LeTian

Tel: 13467509790

---

From: Benny Halevy <[email-masked]>
Date: Mon, 30 Jun 2008 08:43:03 +0300

On Jun. 29, 2008, 17:32 +0300, "???" <[email-masked]> wrote:
> 
> 
> 2008/6/29, Benny Halevy <[email-masked] <mailto:[email-masked]>>:
> 
>     Yi,
> 
> with parity, and client must cach data to compute parity), but
> RAID-0,1... ? i do not think RAID-0,1 need to lock whole stripe..

Indeed, RAID-0 (which isn't really RAID as for redundancy...) need no
locking. RAID-1, in this particular case, where the clients write into
disjoint areas need no locking, however once the server has given out
the layouts it has no guarantee that the client writes won't overlap.

>   while doing "pre-allocate"(do not only one block be allocated), i
> think if the write layouts overlapping a block is granted to two
> clients, and the server can perform LAYOUTCOMMIT, the data may be
> corruption. for example, A CLIENT allocate new blocks (file logic
> address 1M-2M), and B CLIENT get the layout of 1M-2M (of course, both
> clients doing "write only" or "wirte_read" file), and the A CLIENT
> commit the file of size 1.5M, the B CLIENT will write the range of
> 1.5M-2M range of the file but at this time these block may be allocated
> to other files.

That's definitely true to be on the safe side.  I think there could be
one exception, as I mentioned above, if the server supports committing
layouts with single byte granularity it can, in theory, correctly merge
LAYOUTCOMMITs that overlap a block by copying data out of provisionally
allocated blocks rather than committing then into the file, if the respective
byte-ranges are already allocated to the file and the client LAYOUTCOMMITs
only a part of the block.

---

From: "Noveck, Dave" <[email-masked]>
Date: Mon, 30 Jun 2008 09:48:19 -0400

I agree with what Benny is saying but I would organize things a bit
differently and I hope that would make things a bit clearer.  Anyway,
having people come at things from different angles can often help to
elucidate the structure of things.

I think the prime point is that pNFS, in and of itself, does not do
anything relevant to locking.  Basically, that remains as it was in
v4.0.  If an application is so structured that multiple clients will be
writing to a file at the same time, the application either has to have
to be extraordinarily careful, or it has to get byte-range locks.  If
your client code rounds IO's up or down to block boundaries you have to
take this into account and also need to flush dirty data before you
release your locks.  For better or worse, this is how NFS has always
worked and pNFS doesn't change that.  The parallelism added by pNFS is
parallelism among data servers.  Parallelism among clients has to be
managed as it always has been.

Concurrency restrictions on layouts, which give them somewhat the
character of locks, are only applied by metadata servers when there is
something about the data protocol that makes such concurrency
restrictions necessary.  If there isn't such a need, as there isn't when
NFSv4.1 itself is used as the data protocol, then layouts can be given
out quite liberally, with no concurrency restrictions, and only recalled
if there is a change to a file's striping pattern, and so, very rarely.

When there are specific issues, such as the ones Benny has been
discussing, which prevent other data protocols, from providing the
service that NFSv4.1 as a data protocol can, will there be a need to
restrict layout concurrency, as has been discussed.

The important point here is that clients and application can only rely
on this sort of "locking" when it covers some issue in the data
protocol.  If an application assumes it will always be present, and does
not do the appropriate locking at the application level, then you are
going to have an application which is hard to move to another
environment and may not be reliable anyway.  pNFS layouts and the data
protocols that they use are optional, and clients always have to be
prepared to write to the metadata server instead of using the layout.
If you are depending on concurrency restrictions to provide
application-level synchronization support, then there will be a massive
back door in the form of IO directly to the metadata server, that can
undercut that.

Applications and clients need to be written assuming they will be
writing directly to the metadata server and that having multiple data
servers to write to is an optimization.  If you do whatever locking
would be necessary in the framework of writing directly to the metadata
server, then application level issues are covered.  The metadata server
itself is responsible for implementing whatever additional locking is
made necessary by the choice of data protocol.

-----Original Message-----
From: Benny Halevy [mailto:[email-masked]] 
Sent: Monday, June 30, 2008 1:43 AM
To: ???
Cc: [email-masked]
Subject: Re: [nfsv4] question about pNFS.

On Jun. 29, 2008, 17:32 +0300, "???" <[email-masked]> wrote:
>   as to the lock of whole stripe, do you means the RAID-5,6...(all 
> RAID with parity, and client must cach data to compute parity), but 

> 1.5M-2M range of the file but at this time these block may be 
> allocated to other files.

That's definitely true to be on the safe side.  I think there could be
one exception, as I mentioned above, if the server supports committing
layouts with single byte granularity it can, in theory, correctly merge
LAYOUTCOMMITs that overlap a block by copying data out of provisionally
allocated blocks rather than committing then into the file, if the
respective byte-ranges are already allocated to the file and the client
LAYOUTCOMMITs only a part of the block.