Subject: [nfsv4] Re: review osd pnfs draft

From: Benny Halevy <[email-masked]>
Date: Wed, 16 Jan 2008 12:50:52 +0200

On Jan. 14, 2008, 18:54 +0200, Pete Wyckoff <[email-masked]> wrote:
> I'm not going to FAST either.  But your discussion pushed me to
> finish the last few pages of the NFSv4 pNFS OSD spec.  What's the
> timescale for comments/revisions etc?

I'm almost ready with draft-05 of pnfs-obj, however last call for nfsv4.1 I-D
has already started so I don't see anything substantial changing in the protocol.

> 
> I'll jot down some of the bigger items that I ran into, and see what
> your initial comments are.  Feel free to CC this to others if
> necessary.

Thanks much for your comments!

> 
> 1.  Non-OSD specific issues
> 
> Many sections seem general to mulitple types of storage servers, in
> particular block and file.  There is no special requirement for OSD.
> Why is the discussion of these aspects in the OSD-particular spec?
> I'd think a raid5 approach to block would be quite useful to some,
> as would the two-dimensional mapping scheme.
> 
> 	- 2.1 target naming conventions
> 	- 3.2 data mapping
> 	- 3.3 more raid algorithms

I agree with your point.  Initially I suggested a very similar approach
where all layout types share the same high-level mapping data structures
so you can map a (logical) file onto one or more extents where each extent
is a block range in a LUN/volume/object/file on the Data server, and then
the upper layer can do concatenation/mirroring/striping/raid/etc. over
the extents, but this approach was not accepted.  Instead each layout type
specifies its own mapping structures and only the objects layout type
supports client-based raid.  I guess that the main reasons for making
this decision were practical, to allow parallel development of the layout
type specifications, and some related to existing know-how and implementation
experience of the parties involved.

I hope to live to see that changes in nfsv4.2 ;-)

> 
> 
> 2.  More precise implementation description
> 
> The spec should give an implementor all the necessary information to
> write a compliant server or client (or both).  The client side is
> probably easy enough to figure out, but I would like it to be a bit
> clearer.  Issues are scattered here and there through the document,
> like 4.1.1 happens to mention that all WRITEs must have a GETATTR
> for the object used capacity.  Rather subtle.  

OK.

> And when does a
> client need to FLUSH, and data or metadata or both?  

Timing of flush is part of the nfs client's job and that's closely
related to the semantics it provides to its operating environment users.
E.g. Posix clients would flush on fsync, or memory pressure evicting
dirty pages out of the page cache.  Since we're talking about NFS,
flush would be done whenever an NFS COMMIT RPC would have been sent
to the metadata server.  Note that there's also a requirement for
sending LAYOUTCOMMIT after flushing to make sure the newly written
data is committed into the file's layout, and that the file size
and other attributes are updated.

> Can you call
> out exactly the steps a client may have to do, and express them as
> OSD commands:  NOSEC GETATTR sysid, READ (incl short read result
> possibility), WRITE + GETATTR, FLUSH, etc.

Good idea.

> 
> Regarding a server implementation, there's lots of information
> missing.  Perhaps your goal is not to define how a pNFS server
> talks to an OSD, and that's fine if that is the case.

Correct.  This document's scope is the layout-type protocol details
as well as OSD specific considerations for the client.
Server to OSD details are mostly out of scope of NFSv4.1.

> 
> 
> 3.  Parallel client file sharing
> 
> One interesting application for pNFS is in high-performance
> computing and parallel I/O, where a set of cooperating processes
> (via MPI, e.g.) all write data into the same file at different
> non-overlapping offsets.  There are some bits in this OSD spec that
> may make that difficult.
> 
> Sec 3.3.5 says you cannot do RAID in this situation.  Can we instead
> let the clients figure out how to implement that if they wish?
> Maybe run a recalculation from one task at file close time?

Hmm, can you please quote exactly where you got that impression from?
The test in draft-04 says:
   RAID layouts with redundant data in their stripes require additional
   serialization of updates to ensure correct operation.  Otherwise, if
   two clients simultaneously write to the same logical range of an
   object, the result could include different data in the same ranges of
   mirrored tuples, or corrupt parity information.  It is the
   responsibility of the metadata server to enforce serialization
   requirements such as this.  For example, the metadata server may do
   so by not granting overlapping write layouts within mirrored objects.

This doesn't mean RAID can't be used, but rather that the server may
need to serialize read/write layouts for overlapping stripes so that
only one client can write to any specific stripe at a time.

In any case, this is positioned as a recommendation. I can imagine that
with cooperating clients, servers can be configured to allow the clients
to step on each other's toes if the wish to (or hopefully not if they can).

> 
> Sec 6 mentions how capabilities for different parts of a layout
> cannot be merged.  Then later in 9.2, it claims that caps can be
> made byte-range specific in OSD-2.  I can't find this in the latest
> OSD spec.

Correct.  We're still working on that.  The text in 9.2 is anticipative.
I will will add a note in the next draft about that though so it's
clear that this matter is not reflected in the OSDv2 draft.

> 
> But my feeling is that we don't want to do byte-range assignments
> for capabilities used by cooperating clients.  We want to "delegate"
> the entire layout to a set of processes, give them all a layout that
> works on the entire set of objects, and let them figure it out.  Is
> this possible in the pNFS OSD spec?

Absolutely.  The server decides what layout ranges to give and it can
be implemented to support larger (or full file) layouts.

Note that in case the client is doing RAID over the objects the
server needs to clean up after it if the client goes away without
successfully returning the layout and handing out bigger writable
layout means recovering more data which may take considerably longer
(think of multi-terabyte files...).  That might not matter to some
government or academic accounts though ;-)

> 
> 
> Let me know what you think on the above ideas.  I have some smaller
> issues that may or may not be relevant after that.
> 
> 		-- Pete

Keep 'em coming :)
your comments are on the spot and are very valuable!

Thanks again,

Benny

_______________________________________________
nfsv4 mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/nfsv4