Subject: Re: [nfsv4] New version of sparse draft(draft-hildebrand-nfsv4-read-sparse-01.txt)

From: "Matt W. Benjamin" <[email-masked]>
Date: Sun, 3 Oct 2010 11:20:29 -0400 (EDT)

Hi Pranoop,

I noted myself that the next offset return requirement. was potentially onerous, for the reason you state.  However, the response from Dean and explicated by Bruce is that the DS was not meant to be obligated to return the next filled offset, merely to return the information it has.  (Yes, you've raised an interesting point about this.)

It still seems to me, offhand, just as problematic for a different subset of implementations to have a new requirement to propagate hole information (synchronously?) to the MDS, as to have the DSes collectively aware of it.  That is, is it not a potential advantage of pNFS, as currently specified, that the MDS need not have global hole information either?

Thanks,

Matt

----- "Pranoop Erasani" <[email-masked]> wrote:

> > Hi,
> > 
> > Just relative to pNFS, my immediate reaction was that a DS might
> have
> > the relevant file allocation information, and the MDS might not. 
> 
> DS might have relevant file allocation information on data on itself,
> 
> not on other DSs'. AFAIK, pNFS does not mandate that each DS know
> about other DSs' information. It's the MDS that has access to DS
> information.
> 
> Given the current proposal, as part of the READ response, the DS is
> supposed to
> send the offset where the hole ends and the actual data begins. If the
> hole ends
> on a different DS, how is that DS responding to READ supposed to know
> this.
> How does the data server get to compute the information? Are all DSs'
> conscious
> Of how data is organized on other DSs' or is it the duty of the MDS to
> own
> that information? This is where, I feel that the existing proposal
> puts onerous 
> requirements on the pNFS Data Servers.
> 
> > With
> > READZ (is that the current operation name?), this doesn't seem to
> > present a problem. 
> 
> The draft seems to addresses the problem by just mentioning that each
> DS needs
> to know other DS's information.

This was clarified in subsequent email, as I attempt to summarize above.

> 
>    receives.  In addition, when a data server is returning a
>    READ4reshole structure, it should still contain the offset and
> length
>    of the next allocated block in the file, even if that block is not
>    located on that particular data server.
> 
> However, that is not a requirement that pNFS puts on the data servers.
> Isn't it? Are
> we wading into a path of new requirements for pNFS data servers here?
> 
> If my suspicion is true, the spirit of this proposal would discourage
> pNFS servers
> from implementing the sparse hints.
> 
> > It would, I guess, perhaps also not present a
> > problem if clients could get a hole map from a DS, but I think
> that's
> > not what the prior email seemed to be describing?
> 
> Well.. I started with the fact that some vendors could consider that
> hole map
> could be metadata  and thus implied that MDS would be in a better
> position to
> serve that rather than individual data servers (especially, if the
> holes
> span data servers).
> 
> To address the pNFS specific concerns from my original e-mail, we need
> to answer:
> 
> 1). Who owns the hole information
> 2). Who sends the hole information
> 3). How efficiently, they can communicate hole information spanning
> pNFS server set
> 
-- 

Matt Benjamin

The Linux Box
206 South Fifth Ave. Suite 150
Ann Arbor, MI  48104

http://linuxbox.com

tel. 734-761-4689
fax. 734-769-8938
cel. 734-216-5309

---

From: "Erasani, Pranoop" <[email-masked]>
Date: Sun, 3 Oct 2010 11:29:17 -0700

> -----Original Message-----
> From: Matt W. Benjamin [mailto:[email-masked]]
> Sent: Sunday, October 03, 2010 8:20 AM
> To: Erasani, Pranoop
> Cc: Benny Halevy; [email-masked]; J. Bruce Fields
> Subject: Re: [nfsv4] New version of sparse draft(draft-hildebrand-
> nfsv4-read-sparse-01.txt)
> 
> Hi Pranoop,
> 
> I noted myself that the next offset return requirement. was potentially
> onerous, for the reason you state.  However, the response from Dean and
> explicated by Bruce is that the DS was not meant to be obligated to
> return the next filled offset, merely to return the information it has.
> (Yes, you've raised an interesting point about this.)

Yup, if pNFS data servers are not obliged to return this, there is not
much value w.r.t pNFS.

> It still seems to me, offhand, just as problematic for a different
> subset of implementations to have a new requirement to propagate hole
> information (synchronously?) to the MDS, as to have the DSes
> collectively aware of it.  That is, is it not a potential advantage of
> pNFS, as currently specified, that the MDS need not have global hole
> information either?

I totally agree. I was not requiring that all MDSs' have information
about all DSs' holes. My wording initially may gave given some
hint as to this requirement, but the intent was not to require. But, to
state that our server could potentially have the information.

But, in general, an MDS is more likely to have information about DS than
a DS about another DS.

- Pranoop

---

From: Thomas Haynes <[email-masked]>
Date: Sun, 3 Oct 2010 21:54:00 -0500

The harder case is for a non-clustered pNFS community, i.e., one
where the MDS and DSes need a control protocol to talk. Let's
assume that is the case here.

A hole is created when a WRITE happens at a DS that is not
contiguous with the previous (if any) WRITE.

If a DS is supposed to return the range until the next non-zero
data, then it will need to contact every other DS or perhaps
the MDS.

If the MDS is supposed to honor a GET_SPARSE_BLOCK_MAP
type of request from the client, then either each DS would have
to send hole entries to the MDS or the MDS would have to
probe each one.

What if instead of a DS returning information about other DSes,
it simply returns information about what it knows? I.e., my next
data is at address X.

Consider a file with a width of 32k. The first write it gets is at 1M
and then it gets 5 other contiguous writes. The next data is at
2M and it gets 4 other contiguous writes. The DS can't assume
that any other DS has zeros in the first 1M. Worst case, all other
N-1 DSes do have data there.

In that scenario, N-1 DSes will return 32k on the first read and
1 DS will return a hole for it only until 1M. The client will then
only send N-1 READs until it it gets to the 1M mark.

---

From: Benny Halevy <[email-masked]>
Date: Mon, 4 Oct 2010 10:55:22 -0400

> it simply returns information about what it knows? I.e., my next

I think that this is much cleaner from the protocol perspective.
The DS is essentially serving NFSv4.1 so its READ result applies
to the current filehandle which represents the slice stored
on it, not the logical file striped across multiple DSs.

Benny

> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4
>

---

From: <[email-masked]>
Date: Mon, 4 Oct 2010 13:29:09 -0400

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Benny Halevy
> Sent: Monday, October 04, 2010 7:55 AM
> To: Thomas Haynes
> Cc: J. Bruce Fields; Erasani, Pranoop; [email-masked]
> Subject: Re: [nfsv4] New version of sparse 
> draft(draft-hildebrand-nfsv4-read-sparse-01.txt)
> 

I like this approach as well.  However it does present some complexity.  The value returned to the client has to be interpreted by the client, using the layout, to determine whether data may exist on another DS in the _possible_ hole.  Dense vs. sparse striping must also be taken into account.  The DS can only return the next byte offset in the stripe file where data exists (having no knowledge of stripe boundaries or dense/sparse striping).

  -Dan