Subject: Re: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5667bis-04.txt

From: Chuck Lever <[email-masked]>
Date: Wed, 1 Feb 2017 12:01:38 -0500

> On Jan 31, 2017, at 7:59 PM, David Noveck <[email-masked]> wrote:
> 
> > I've attempted to make the language more generic, but I
> > regard that as a constructive but ultimately unattainable
> > goal because it requires predicting the future with a high
> > degree of accuracy.
> 
> I don't agree.

I think you do agree; as you said below, the issue is what
are the actual completion criteria.

> I think we need use more abstraction in
> stating the requirements than rfc5667 did.

Obviously I agree with that, since there already is more
abstraction in rfc5667bis than there was in RFC 5667, and
I have stated that the work isn't done yet.

> The NFS specs
> work with filesystems that were not known when the
> spec were written.  Only when we extend the 
> abstraction, as we did with xattrs, do we have to 
> write a new extension document.
> 
> > At some point we have to stop and say it is good enough.
> 
> We agree on that but disagree about what that point is.
> 
> 
> > > The concept of an
> > > extensible version 2 transport would become unworkable in that
> > > case.
> 
> > IMO that's overstating it. We have a lot to learn about how to
> extend RPC-over-RDMA. 
> 
> > The extension rules in rpcrdma-version-two
> > have to state how extension specifications deal with adding
> > new ULB requirements. 
> 
> I don;t think extensions need to add ULB requirements.

I mean the converse: An extension _may_ add ULB requirements.
Thus there have to be guidelines written somewhere about how
that is accomplished, even if they might not be immediately
used.

> I do think we
> need some text in rpcrdma-version-two that explains what the extensions
> have to say if they propose different implementation of abstractions
> referred to by ULB(s).

That's also fine.

> I'll draft some proposed text in this regard and run it by you.
> 
> > Right now, we don't know what requirements
> > even existing proposed extensions might add. For instance:
> 
> > - Do we really believe that rfc5666bis DDP eligibility is an
> > appropriate rule to apply, without any change, to send-based DDP?
> 
> I believe they do.  rtrext is written using that concept which
> you created and required that ULBs define.  That gives us a good
> basis to make ULBs more generic.  I just want to take
> advantage of it sooner than you do.

You want to take advantage of it before these extensions are
finished, before some have even been reviewed and discussed
publicly. That is what I object to.

As previously stated, I don't have a problem making rfc5667bis
more generic. But care must be taken. It can go too far, and
it especially isn't a good idea to push this generality based
on protocol designs that are not mature.

For instance, as to the technical issue about whether DDP
eligibility is appropriate for send-based DDP without change:
I believe it isn't.

1. The rfc5666bis definition of DDP eligibility makes sense
if you are dealing with a mechanism that is more efficient
with large payloads than small. Send-based DDP is just the
opposite.

2. Re-assembling an RPC message that has been split by
message continuation or send-based DDP is done below the XDR
layer, since all message data is always available once Receive
is complete, and XDR decoding is completely ordered with the
receipt of the RPC message. With explicit RDMA, receipt of
some arguments MAY be deferred until after XDR decoding
begins.

In fact, the current RPC-over-RDMA protocol is designed
around that. For example the mechanism of reduction relies on
only whole data items being excised to ensure XDR round-up of
variable-length data types is correct. That is because an
implementation MAY perform RDMA Read in the XDR layer, not
in the transport.

3. Message continuation, like a Long message, enables an RPC
message payload to be split arbitrarily, without any regard
to DDP eligibility or data item boundaries.

4. Send-based DDP is not an offload transfer. It involves the
host CPUs on both ends. Thus it doesn't make sense to restrict
the DDP eligibility of data items that require XDR marshaling
and unmarshaling.

A receiver has generic receive resources that have to
accommodate any RPC that is received (via Send/Receive). DDP
eligibility is designed to prepare the receiver for exactly
one particular data item that MAY involve the ULP in the
reconstruction of the whole RPC message.

Explicit RDMA and send-based DDP are perhaps in the same
phylum but are truly distinct beasts.

> > I don't think that's a foregone conclusion.
> 
> Perhaps not but I believe it is the case.  
> 
> > - Can you explain here why you believe that message continuation
> > will have an impact on the way ULBs are written?
> 
> Right now rfc5667 says that if a message is longer than the
> size of the receivers buffer it must go in a reply chunk.  That
> isn't compatible with message continuation.
> 
> Instead, it should say that if a message is longer than the longest 
> message that a transport can handle using sends, it should be 
> transferred using explicit RDMA operations.

That is a clear, simple, and actionable review comment,
unlike "rfc5667bis forecloses on the use of message
continuation".

This specific issue wasn't mentioned in your previous
comments about this I-D, so there was no way for me to
address it before.

An alternate way to address the comment is to remove that
kind of language from rfc5667bis, since the general
topic of Long messages is already covered by rfc5666bis.
I'll go have a look.

> That would leave it up to the transport to define the longest message 
> that can transferred by sends and how longer messages are transferred
> using explicit RDMA operations, which rfc5666bis already does.
> 
> > The point being that we have a set of ULB composition rules in
> > rfc5666bis. You seem to be adding new ones that we haven't all
> > agreed on, and then dinging me (repeatedly) for ignoring them.
> 
> I haven't done that.  If it seems to you that I have, it would help
> if you specified one of these new rules.

> > I'm much more comfortable closing the book on rfc5667bis with the
> > set of transports we have in RFCs now, and then opening another
> > update when we have real documented transport changes to address.
> 
> If you want to close the book on rfc56667bis, I think I can put
> off what I was planning to do.
> 
> > Otherwise rfc5667bis will never be finished.
> 
> I think you are unduly pessimistic but I think we accommodate your
> desire to close on rfc5667.  I'll send separate mail on the retry issue.

It is not pessimism. It is that the targets are moving, and
not fully documented: a recognizable pattern for creeping
scopes and slipping delivery schedules.

> > Let's design for what we have, 
> 
> OK.  rfc5667bis will address what we have. limited to protocols for which there
> are implementations, and working group documents that are far advanced.
> 
> Another document will address what we have in the sense that there are existing specifications 
> for the feature.  In other words it will hande version-two and rtrext.  I think it will handle a lot 
> more, although you might not agree.
> 
> > clear the table, and move on. There's
> > no reason to believe that will create an unending sequence of updates
> > to the NFS ULB. There will be no more ULB updates than there are
> > extensions, and quite likely fewer.
> 
> Exactly.  I think we can get by with exactly one.
> 
> So what I expect to do is to submit draft-dnoveck-nfsv4-nfsulb, illustrating my
> approach to making the ULBs for NFS transport-generic.  Since it is based
> on rfc5667bis, I will, with your permission, list you as a co-author.

I would rather wait until it is clear that rfc5667bis cannot
accommodate your goals, and focus for now on:

1. continued iterative review of rfc5667bis, and

2. composing language for rpcrdma-version-two that provides
guidelines for ULB updates in extensions.

> On Tue, Jan 31, 2017 at 6:02 PM, Chuck Lever <[email-masked]> wrote:
> 
> > On Jan 31, 2017, at 2:54 PM, David Noveck <[email-masked]> wrote:
> >
> > > Thus I'm considering removing the retry language from rfc5667bis.
> >
> > I don't think that should be done.  Let me explain why.
> >
> > > First, the mandate I recall receiving in Dallas for rfc5667bis was:
> > >
> > > 1. Document existing implementations
> > > 2. Fix mistakes and omissions
> > > 3. Extend the NFS ULB to properly cover NFSv4.1 and NFSv4.2
> >
> > And those have been done :-)
> >
> > > Since then we have added:
> >
> > > 4. Make it align with new language in rfc5666bis
> >
> > One part of which is to require that you have a reliable means of
> > bounding reply sizes.  Allowing retry is a means to accomplish
> > that requirement without essentially waving the problem away
> > and guessing/hoping that it will not be bothersome.   Even if this
> > is not a problem in practical terms, I don;t see how we can present
> > an rfc5667bis that doesn't meet the requirements in rfc5666bis.
> >
> > > And snuck in:
> >
> > I don't think there was anything sneaky here.  The "mandate"
> > you mention was not a legislative act or executive order.
> > It was a plan to go forward with and we should be able
> > to make changes if necessary.
> 
> > > 5. Make it align with the language of rpcrdma-version-two and
> > > rpcrdma-rtrext (which I'm also struggling with, but that's a
> > > separate topic)
> >
> > I think the goal is not to align it with the particulars of any
> > particular version or extension but to make it generic, so
> > that the transport can be changed without continually revising
> > the ULB for new versions and extensions.
> 
> I've attempted to make the language more generic, but I
> regard that as a constructive but ultimately unattainable
> goal because it requires predicting the future with a high
> degree of accuracy.
> 
> At some point we have to stop and say it is good enough.
> 
> 
> > The concept of an
> > extensible version 2 transport would become unworkable in that
> > case.
> 
> IMO that's overstating it. We have a lot to learn about how to
> extend RPC-over-RDMA. The extension rules in rpcrdma-version-two
> have to state how extension specifications deal with adding
> new ULB requirements. Right now, we don't know what requirements
> even existing proposed extensions might add. For instance:
> 
> - Do we really believe that rfc5666bis DDP eligibility is an
> appropriate rule to apply, without any change, to send-based DDP?
> I don't think that's a foregone conclusion.
> 
> - Can you explain here why you believe that message continuation
> will have an impact on the way ULBs are written?
> 
> The point being that we have a set of ULB composition rules in
> rfc5666bis. You seem to be adding new ones that we haven't all
> agreed on, and then dinging me (repeatedly) for ignoring them.
> 
> 
> > As an example, the current rfc5667bis, in numerous places
> > assumes that no mesaage continuation feature is available.  We have
> > suffered, for a long time, from the lack of this feature which SMB
> > Direct has had for a long time.  Even apart from the controversial
> > concept of send-based DDP, I don't see how we can have an rfc5667bis
> > that essentially forecloses a message continuation extension.
> 
> I'm much more comfortable closing the book on rfc5667bis with the
> set of transports we have in RFCs now, and then opening another
> update when we have real documented transport changes to address.
> Otherwise rfc5667bis will never be finished.
> 
> Let's design for what we have, clear the table, and move on. There's
> no reason to believe that will create an unending sequence of updates
> to the NFS ULB. There will be no more ULB updates than there are
> extensions, and quite likely fewer.
> 
> 
> > > As I see it, "retrying short Reply chunks" is a new and untried
> > > piece of protocol. It attempts to accommodate a future,
> > > unimplemented transport or ULP, and therefore it lies outside the
> > > mandated scope of rfc5667bis IMO.
> >
> > But that position would essentially ties us to Version One forever, which,
> > aside from the particulars of the discussion in Dallas, is not where we want
> > to be.,
> 
> It doesn't tie anyone to anything. The WG is allowed to open
> updates of existing RFCs as many times as they like. I believe
> there is adequate process here to deal properly with it.
> 
> 
> > I think Version One can accommodate retry, even though it
> > has some weaknesses with regard to non-idempotent operations.
> 
> ERR_CHUNK can be returned for a lot of reasons. As we learned
> when trying to add extensibility to V1, that could make it
> inadequate for this purpose.
> 
> If short Reply chunk retry is mentioned in rfc5667bis, I prefer
> that it is discussed in the following terms:
> 
> 1. It is an optional remedy, written as implementation advice.
> 
> 2. The choice to retry is at the discretion of the ULP on the
> requester (not the transport)
> 
> 3. Retry is done via a distinct RPC (fresh XID)
> 
> 4. Short Reply chunk retry is permitted only for NFSACL GETACL
> and NFSv4.0 GETATTR
> 
> 
> > It is true that version Two will probably have better, more complete
> > support.
> 
> Possibly all that is necessary is an unambiguos error code.
> 
> 
> > It may be that there is something in rfc566bis as currently written, that
> > makes this difficult to do at this late stage.  I could live with that,
> > but given that rfc5667bis needs to be made transport-generic, I think
> > it should definitely allow this in future transport Versions, if those
> > versions support it, and not forclose it, just as it should not foreclose
> > message continuation.
> 
> Another way to "allow retry in future versions" is not to
> mention it at all, which is what I proposed in the previous
> e-mail.
> 
> 
> > > Second, it's unnecessary in the real world:
> >
> > I agree that it is very unlikely to be used and that many implementations
> > may choose not to implement it.
> >
> > > I'm not aware of a single instance in the field where a server has
> > > had to reject a client's request because of a short Reply chunk.
> > > It's certainly never occurred during my testing, but lots of other
> > > issues have.
> >
> > OK, but the problem is that, without it, when and if this happens,
> > we have a situation in which it is clear there is a bug, but it cannot be
> > determined whether the fault is due to the requester or the responder.
> 
> The responder reports, via a local system log, that the requester
> is in error. The Solaris NFS/RDMA server implementation reports
> similar errors this way all the time, in addition to sending
> GARBAGE_ARGS replies, so I know this works.
> 
> It can be done with V1 implementations, and it can be done even
> if requesters do not support retrying.
> 
> 
> > In order to allow interoperable implementations, this kind of situation
> > must be avoided.
> 
> > If the spec allows retry but the requester implementation does not provide it,
> > then it has a bug and it can decode it is willing to live with a small possibility of
> > failure.
> >
> > >Third, the places where retry might be useful are all legacy
> > > protocols:
> >
> > If you use the term "legacy protocols" as it is used in rfc5667bis, this is not so.
> 
> To be clear, I meant legacy in the sense that the book is closed
> on NFSACL and NFSv4.0.
> 
> 
> > Also, rfc5666bis requires reliable reply estimation of all ULPs, rather than
> > only of non-legacy ones.
> 
> The intent of that requirement is to ensure interoperation.
> 
> You haven't explained why "failing a request that is not formed
> according to the rules" is not sufficient to establish that
> implementations interoperate successfully.
> 
> If the rfc5666bis requirement is too stern or hazy or in some
> other way not sensible, we should fix it.
> 
> But I don't see anything that states a ULB is prevented from
> identifying gray areas or possible cases that can't be handled.
> 
> 
> > On Tue, Jan 31, 2017 at 12:04 PM, Chuck Lever <[email-masked]> wrote:
> >
> > > On Jan 31, 2017, at 7:11 AM, David Noveck <[email-masked]> wrote:
> > >
> > > > Without an RPC Reply message, however, the client matches the XID in the
> > > > ERR_CHUNK message to a previous call and that will have the matching
> > > > SEQUENCE operation.
> > >
> > > Right.
> >
> > As I clicked Send yesterday, I blinked and "retried" became "retired".
> > Although I'm a fan of "retired", a different term might be less likely
> > to confuse readers in this context. Suggestions welcome.
> >
> > So now I read this text
> >
> > >    In addition, within the error response, the requester does not have
> > >    the result of the execution of the SEQUENCE operation, which
> > >    identifies the session, slot, and sequence id for the request which
> > >    has failed.  The xid associated with the request, obtained from the
> > >    rdma_xid field of the RDMA_ERROR or RDMA_MSG message, must be used to
> > >    determine the session and slot for the request which failed, and the
> > >    slot must be properly retired.  If this is not done, the slot could
> > >    be rendered permanently unavailable.
> >
> > to mean "No short Reply chunk retries are permitted when a session is
> > in use." I'm much more comfortable with this requirement.
> >
> > And, now there is one less use case for retrying a short Reply chunk.
> >
> >
> > > > That makes this rather a layering violation,
> > >
> > > It looks to me like this layering violation is not serious and I think it is
> > > unavoidable.
> >
> > Let's consider how the layers might work together in this case.
> >
> > Sizing a Reply chunk is done by the ULP. If the Reply chunk turns out
> > to be too small, it is the ULP, not the transport, that will have to
> > replace that chunk with a larger Reply chunk.
> >
> > The ULP therefore must be fully aware that the previous attempt
> > failed, and why. It also has to be capable of deciding whether or not
> > a retry is advisable, given other factors such as responder reply
> > caching.
> >
> > Since the ULP is driving the retry, given the separation of duties
> > between a ULP and the RPC layer, it is possible that a new XID will
> > be used for the retried operation. The responder does not have a way
> > of recognizing the link between the failed XID and the new one.
> >
> > To address this, then the ULP would have to indicate what XID should
> > be used in the retry's RPC (and RPC-over-RDMA) header.
> >
> >
> > > > and perhaps a reason why
> > > > retransmitting with a larger Reply chunk might be a cure worse than the
> > > > disease.
> > >
> > > Note that the layering violation does not arise when retrying the operation
> > > with a larger reply chunk.
> > >
> > > The case when this layering violation/misdemeanor occurs is when
> > > the operation is failed and the slot needs to be made available again.  Retrying
> > > the operation with a larger reply chunk would make make this situation.
> > > less likely to occur. However, since 4.1 has a reliable means of limiting reply
> > > size (unlike v4.0), it appears that this is beside the point for session-based
> > > minor versions of nfsv4.
> > >
> > > Note that the disease here is the fact that v4.0  (and some auxiliary
> > > protocols) does not have a reliable means of determining reply size limits.
> > > We can't cure that disease as these protocols are unchangeable.
> > >
> > > Allowing retry with a larger reply chunk is not a cure but it is a treatment which
> > > ameliorates the problem.  As far as I can see, it is a safe and effective treatment.
> >
> > The safe and most conservative course is to terminate the RPC, in all
> > cases. Some ULPs might be able to tolerate retry, others might not.
> >
> > Each ULP has to make that decision on a case-by-case basis. The
> > transport by itself cannot arbitrarily perform a retry, it is now
> > clear.
> >
> > Thus I'm considering removing the retry language from rfc5667bis.
> >
> > First, the mandate I recall receiving in Dallas for rfc5667bis was:
> >
> > 1. Document existing implementations
> > 2. Fix mistakes and omissions
> > 3. Extend the NFS ULB to properly cover NFSv4.1 and NFSv4.2
> >
> > Since then we have added:
> >
> > 4. Make it align with new language in rfc5666bis
> >
> > And snuck in:
> >
> > 5. Make it align with the language of rpcrdma-version-two and
> > rpcrdma-rtrext (which I'm also struggling with, but that's a
> > separate topic)
> >
> > As I see it, "retrying short Reply chunks" is a new and untried
> > piece of protocol. It attempts to accommodate a future,
> > unimplemented transport or ULP, and therefore it lies outside the
> > mandated scope of rfc5667bis IMO.
> >
> > Second, it's unnecessary in the real world:
> >
> > I'm not aware of a single instance in the field where a server has
> > had to reject a client's request because of a short Reply chunk.
> > It's certainly never occurred during my testing, but lots of other
> > issues have.
> >
> > Third, the places where retry might be useful are all legacy
> > protocols:
> >
> > As you pointed out, NFSv4.0 and NFSACL appear to be the two areas
> > where we have concerns. The problem is largely addressed by NFSv4.1
> > and newer minor versions, and even there, retrying is not permitted.
> > The future does not need this new behavior, apparently.
> >
> > What retrying amounts to, in this context, is adding a workaround
> > in the transport protocol for bugs in the Upper Layer Protocols and
> > their implementations.
> >
> >
> > > > I'm beginning to believe that making this situation always a permanent
> > > > error, as rfc5666bis does, is a better protocol choice.
> > >
> > > I don;t see it that way.  It leaves us with an rfc5666bis  requirement for ULBs that
> > > we would be unable to satisfy for a number of ULPs dealt with in rfc5667bis.
> >
> > This is a hole that can be closed simply by prescribing how each
> > implementation must behave when a Reply chunk is short. That has
> > already been done in rfc5666bis: the RPC fails. This prevents a
> > transport deadlock and connection loss, and indicates a ULP
> > implementation bug.
> >
> > Why does rfc5667bis need to go further? The consequences of a
> > shortage are clear, and no longer catastrophic to other RPCs.
> >
> >
> > --
> > Chuck Lever
> >
> >
> >
> >
> 
> --
> Chuck Lever
> 
> 
> 
> 
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 1 Feb 2017 13:52:33 -0500

> that is accomplished, even if they might not be immediately

I think the best way to handle this is for me to propose changes to
rpcrdma-version-two and let you comment on them.

> That's also fine.

So now I have two items on my to-do list for rpcrdma-version-two.  I'll
probably do them together.

> publicly. That is what I object to.

A couple of points:

   - I want to discuss how these extensions would be used by NFS and it
   appears to me that the simplest way of doing that is for me to publish
   nfsulb.
   - This is particularly important since the only ULBs in existence are
   the ones for NFS.
   - The extensions have been published.  It is true that they haven't had
   the review they should have but I can't make people review things.  I'm not
   proposing that they be made working group documents, let alone saying they
   are ready for WGLC.  They aren't ready but it would help to begin
   discussion of how these would be used in NFS.
   - Publishing nfsulb would allow Chuck to take whichever changes he
   wishes into rfc5667bis and defer/reject those he is uncomfortable with

> 1. continued iterative review of rfc5667bis, and

I 'm OK with that, but I feel that rather than continue an abstract
discussion of the phyletics of various DDP species, I think I can
publish nfsulb as an example of my approach and Chuck and I,
together with the rest of the group, can discuss my proposed changes
 and Chuck can decide whether the are appropriate to rfc5667bis.

> guidelines for ULB updates in extensions.

I'm going to do that.

One remaining issue concerns Chuck potential role as a co-author of
nfsulb.  Based on his comments, I assume that he wouldn't be comfortable
with that and so I intend to instead credit him in the acknowledgements
section. If Chuck wants, we can change that.

---

From: Chuck Lever <[email-masked]>
Date: Wed, 1 Feb 2017 15:14:42 -0500

> On Feb 1, 2017, at 1:52 PM, David Noveck <[email-masked]> wrote:
> 
> So now I have two items on my to-do list for rpcrdma-version-two.  I'll probably do them together.

All of the above is what I had in mind, too.

> 	• I want to discuss how these extensions would be used by NFS and it appears to me that the simplest way of doing that is for me to publish nfsulb.
> 	• This is particularly important since the only ULBs in existence are the ones for NFS.
> 	• The extensions have been published.  It is true that they haven't had the review they should have but I can't make people review things.  I'm not proposing that they be made working group documents, let alone saying they are ready for WGLC.  They aren't ready but it would help to begin discussion of how these would be used in NFS.

I have reviewed this work, but the review became out of date after
Berlin and I haven't had a chance to revisit it to prepare it again
for posting.

> 	• Publishing nfsulb would allow Chuck to take whichever changes he wishes into rfc5667bis and defer/reject those he is uncomfortable with

So would continuing iterative review on rfc5667bis, but OK, I
can do it this way too.

> section. If Chuck wants, we can change that.

Acknowledgement sounds fine.

--
Chuck Lever

---

From: karen deitke <[email-masked]>
Date: Wed, 1 Feb 2017 14:27:52 -0700

On 1/30/2017 5:28 PM, Chuck Lever wrote:
> Hi Karen, thanks for taking time to review.
>
>
>> On Jan 27, 2017, at 6:40 PM, karen deitke <[email-masked]> wrote:
>>
>>
>> Hi Chuck,
>>
>> o Section 2.6, second to last paragraph, typo, priori should be prior.
> "a priori" is Latin for "in advance" (literally, "from the earlier"). I've
> replaced this phrase.
Thanks, I don't speak Latin :-)

>
>
>> o Same section, "Note however that many operations normally considered non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".  This was confusing because prior in the paragraph you     are talking about determine the reply size for variable length replies, which doesn't seem to be the case for WRITE, SETATTR.
> That's correct, WRITE doesn't have a variable length reply. This section
> discusses a possible COMPOUND with WRITE (fixed length result, non-idempotent)
> and GETATTR (variable length result, idempotent). I've clarified it.
ok
>
>
>> o 4.2.1 seoond paragrah would read better if you removed the word "however"
> I was not able to find the word "however" in the second paragraph of S4.2.1.
> There are certain NFS version 4 data items whose size cannot be 	
>
> 	chunk to disambiguate which chunk is associated with which argument 	
> 	estimated by clients reliably, however, because there is no protocol- 	
>
> 	data item. However NFS version 4 server and client implementations 	
> 	specified size limit on these structures. These include:
>
Sorry, it in 4.2:

"There are certain NFS version 4 data items whose size cannot be 
estimated by clients reliably, however, because there is no 
protocol-specified size limit on these structures."

>> "If a client implementation is equipped to recognize that a transport error could mean that it provisioned an inadequately sized Reply chunk, it can retry the operation with a larger Reply chunk.  Otherwise, the client must terminate the RPC transaction."
>>
>> So you are saying for example that if the client sends a request in which it knows that it has "guessed" the maximum response but that guess may be wrong, if it detects the connection dropped for example, it could potentially interpret that as the guess was too small and send again?
> As previously discussed.
>
>
>> 4.3, second from last bullet, "If an READ", should be "If a READ"
> Fixed.
>
>
>> 4.5.  Why does the xid in the rdma header have to be used to get the session info, wouldnt the session info in the sequence_op that was sent in the first place have this info?
> Here's the last paragraph of Section 4.5:
>
> The mention of RDMA_MSG is probably incorrect. In that case, the server
> was able to return a SEQUENCE result, and that should be useable by the
> client.
>
I'm still not following this.  Is the issue that the  client would 
resend, but the seq of the slot is incremented and we wouldn't 
potentially get anything the server had in the slot replay?

Karen
>
> That makes this rather a layering violation, and perhaps a reason why
>> Karen
>> On 1/20/2017 5:27 PM, [email-masked] wrote:
>>> A New Internet-Draft is available from the on-line Internet-Drafts directories.
>>> This draft is a work item of the Network File System Version 4 of the IETF.
>>>
>>>          Title           : Network File System (NFS) Upper Layer Binding To RPC-Over-RDMA
>>>          Author          : Charles Lever
>>> 	Filename        : draft-ietf-nfsv4-rfc5667bis-04.txt
>>> 	Pages           : 18
>>> 	Date            : 2017-01-20
>>>
>>> Abstract:
>>>     This document specifies Upper Layer Bindings of Network File System
>>>     (NFS) protocol versions to RPC-over-RDMA.  Upper Layer Bindings are
>>>     required to enable RPC-based protocols, such as NFS, to use Direct
>>>     Data Placement on RPC-over-RDMA.  This document obsoletes RFC 5667.
>>>
>>>
>>>
>>> The IETF datatracker status page for this draft is:
>>>
>>> https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5667bis/
>>>
>>>
>>> There's also a htmlized version available at:
>>>
>>> https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5667bis-04
>>>
>>>
>>> A diff from the previous version is available at:
>>>
>>> https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5667bis-04
>>>
>>>
>>>
>>> Please note that it may take a couple of minutes from the time of submission
>>> until the htmlized version and diff are available at tools.ietf.org.
>>>
>>> Internet-Drafts are also available by anonymous FTP at:
>>>
>>> ftp://ftp.ietf.org/internet-drafts/
>>>
>>>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 1 Feb 2017 16:47:50 -0500

> On Feb 1, 2017, at 4:27 PM, karen deitke <[email-masked]> wrote:
> 
> 
> 
> "There are certain NFS version 4 data items whose size cannot be estimated by clients reliably, however, because there is no protocol-specified size limit on these structures."

How about this:

Within NFS version 4.0, there are a few variable-length result data
items whose maximum size cannot be estimated by clients reliably
because there is no protocol-specified size limit on these
structures.

> I'm still not following this.  Is the issue that the  client would resend, but the seq of the slot is incremented and we wouldn't potentially get anything the server had in the slot replay?

Is there a way to recover if the server can't send a SEQUENCE result for
that slot? Seems like the slot would be stuck on the same RPC until the
session is destroyed.

>>> On 1/20/2017 5:27 PM, 
>>> [email-masked]
>>>  wrote:

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Wed, 1 Feb 2017 17:04:34 -0500

> because there is no protocol-specified size limit on these

I like this but I have a problem with the word "structures".   The only case
we are talking about is a GETATTR response, and the XDR defines that as an
opaque array, even though the client and server both are supposed to know
what is in it.  "arrays" would work.

On Wed, Feb 1, 2017 at 4:47 PM, Chuck Lever <[email-masked]> wrote:

> >>> o Same section, "Note however that many operations normally considered
> non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".  This was
> confusing because prior in the paragraph you     are talking about
> determine the reply size for variable length replies, which doesn't seem to
> be the case for WRITE, SETATTR.
> >>>
> >>> "If a client implementation is equipped to recognize that a transport
> error could mean that it provisioned an inadequately sized Reply chunk, it
> can retry the operation with a larger Reply chunk.  Otherwise, the client
> must terminate the RPC transaction."
> >>>
> >>> So you are saying for example that if the client sends a request in
> which it knows that it has "guessed" the maximum response but that guess
> may be wrong, if it detects the connection dropped for example, it could
> potentially interpret that as the guess was too small and send again?
> >>>
> >> As previously discussed.
> >>
> >>
> >>
> >>> 4.5.  Why does the xid in the rdma header have to be used to get the
> session info, wouldnt the session info in the sequence_op that was sent in
> the first place have this info?
> >>>
> resend, but the seq of the slot is incremented and we wouldn't potentially
> get anything the server had in the slot replay?
>

---

From: Chuck Lever <[email-masked]>
Date: Wed, 1 Feb 2017 18:04:06 -0500

> On Feb 1, 2017, at 5:04 PM, David Noveck <[email-masked]> wrote:
> 

Agreed, fixed.

This section also mentioned inline thresholds and Reply chunks,
and that looks superfluous. I removed it.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Thu, 2 Feb 2017 08:53:11 -0500

> and that looks superfluous. I removed it.

A whole lot of rfc5667 wound up essentially repeating stuff
in rfc5666.  A lot of that has been removed and that work is
continuing.

The interesting question is when that process should stop.  In
particular, we need to understand if it can stop short of simply
providing the information required of ULBs in rfc5666bis.  When
I suggested going that way, you didn't like the idea and I had
doubts about whether it was workable,  Nevertheless, if we
have material beyond what we need to meet those requirements,
we should understand why it is there.

Meanwhile, in nfsulb, I've been going in the opposite direction,
by leaving in the V1-specific material in and indicating that other
implementations are possible and will be discussed in the documents
for the appropriate versions and extensions.  As a result, while
rfc5666bis is getting smaller, my document is larger than -04.

These documents need to come together at some time, most likely
after rfc5667bis is published (or at least in WGLC) and
rpcrdma-version-two has been turned into a working group
document.

The prospect of doing so brings us back to our deferred discussion
of the phyletics of the various forms of DDP.  My original plans had
been to put this off until nfsulb was published (done), there was a
new version of rtrext that addressed ULB issues (now working on it), and
I added some ULB-related material to rpcrdma-version-two (on my todo
list).

As I looked at your comments about this issue, I still disagreed with them
but decided that they raise issues that the discussion of ULB issues
will have to address, and they might also lead to clarifications in the
rest of rtrext.  So that we don't have to wait for some of these documents

> phylum but are truly distinct beasts.

I agree they are different things.  There would be no point in defining
send-based DDP if was the same as V1-DDP.  Nevertheless, they
serve the same purpose: transferring DDP-eligible items so that the
receiver gets them in an appropriately sized and aligned buffer.  I
thinks that is all that matters for this discussion.

To get back to phyletics, I would put the branching much lower in the
tree.  To me a better analogy is to consider them as two distinct,
closely-related orders, like rodentia and lagomorpha (rabbits,
hares, pikas).

It is true that there are reasons on might choose to move a DDP-eligible
data Item using one or the other but that is an implementation decision.
Efficiency considerations  and  direction of transfer may affect this
implementation decision, as is the fact , mentioned in rtrext, that if the
client wants the response to be placed at a particular address, you
shoulnd't use send-based DDP.  But none of this affects the fact that
only DDP-eligible items can be placed by send-based DDP, just as is the case
with V1-DDP.

> with large payloads than small. Send-based DDP is just the

It depends on what you mean by "large" and "small".
Clearly however, neither technique makes sense for very
small payloads and both mske most sense for payloads
over 4K although the specfic will depend on efficiency and
how the send-based DDP receiver chooses to structure its
buffers.

For me the sweet spot for sen-based DDP would be 8K
transfers and for this reason structured buffers of 9k
(consisting  of 1K primarily for inline payload
together with an 8K aligned segment for direct placement)
are a good choice.   Even larger buffer are possible but the
fact that message continuation can be used means that
send-based DDP should have no problems with 64K
transfers.

it is true that very large transfers might be better using
V1-DDP but that is beside the point.  Any READ buffer is
DDP-eligible (even for a 1-byte READ) and the choice of
whether to use DDP i(or the sort to use) in any particular
case is up to the implementation.  The ULB shouldn't
be affected.

> some arguments MAY be deferred until after XDR decoding

First of all, message continuation isn't relevant to this
discussion.

With regard to send-based DDP, it is true that you have
the placed data immediately, while you might have to wait
in the V1-DDP case, but this doesn't affect  DDP-eligibility.
In either case, before you process the request or response,
all the data, whether inline or placed needs to be available.
I don't see how anything else matters.

> implementation MAY perform RDMA Read in the XDR layer, not

It may be that send-based DDP could have been designed
to accommodate cases that V1-DDP could not but the fact is that
it wasn't because:

   - There are no common cases in which it would provide a benefit.
   - It would complicate the design to make this something other than a
   drop-in replacement for V1-DDP>

> to DDP eligibility or data item boundaries.

Exactly right but this just demonstrates that message
continuation and DDP-eligibility should not affect one
another and they don't.

> host CPUs on both ends.

Not sure what you mean  by that.  Both are the same in that:

   - You don't need to copy data on ether side
   - Both sides CPU"s need to deal with where the data is.  The adapter has
   to be told where the data is to sent/transferred from while the receiving
   side has to note where the placed data is (i.e not in the normal XDR
   stream).

> Thus it doesn't make sense to restrict

This sound like you think coping is required.  It isn't.

There is no point in using send-based DDP to transfer an
item that requires marshalling/unmarshalling and the situation
is the same as  for V1-DDP.  There is no point in specifically
placing data that would get no benefit from being specifically
placed.

> accommodate any RPC that is received (via Send/Receive).

True.

> DDP

True.

On Wed, Feb 1, 2017 at 6:04 PM, Chuck Lever <[email-masked]> wrote:

> > >>> o Same section, "Note however that many operations normally
> considered non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".
> This was confusing because prior in the paragraph you     are talking about
> > >>> "If a client implementation is equipped to recognize that a
> transport error could mean that it provisioned an inadequately sized Reply
> chunk, it can retry the operation with a larger Reply chunk.  Otherwise,
> the client must terminate the RPC transaction."
> > >>>

---

From: Chuck Lever <[email-masked]>
Date: Thu, 2 Feb 2017 15:45:03 -0500

> On Feb 2, 2017, at 8:53 AM, David Noveck <[email-masked]> wrote:
> 
> providing the information required of ULBs in rfc5666bis.

I'm totally OK with removing redundant language from rfc5667bis,
but we should not pretend that DDP eligibility is completely
general or that we understand perfectly how all DDP transport
mechanisms will work in the future.

I'd like to be sure we are not creating a Sisyphean task.

> When 
> doubts about whether it was workable,

For example, some repetition of rfc5666bis language is helpful as
implementation guidance.

> Nevertheless, if we 

There are other things that can go into a ULB that might not
have been listed in rfc5666bis. Clearly one issue we have with
NFS in particular is how to deal with replay caching above the
transport.

> thinks that is all that matters for this discussion.

The numbered items in my previous e-mail demonstrate that the
purpose of DDP eligibility is to encourage ULB and implementation
choices that are appropriate when using offloaded RDMA. They do not
all apply to Send-based DDP, though some do.

Or put another way, DDP eligibility is not general at all! It is
only "general" because Send-based DDP has been adjusted to fit it.

For example, you have argued below that Send-based DDP is aimed at
improving a particular size I/O; yet data item size is one important
criterion for a ULB author to decide whether to make a data item
DDP-eligible.

Suppose a ULP has a 512-byte opaque data item (opaque, meaning that
no XDR encoding or decoding is required) that would benefit from
avoiding a data copy because it is in a frequently-used operation.
You might not choose to make it DDP-eligible because an explicit
RDMA operation is inefficient. But it might be a suitable candidate
for Send-based DDP.

> send-based DDP should have no problems with 64K

With Remote Invalidation, NFS READ can be handled with a Write
chunk with nearly the same efficiency and latency as a mechanism
that uses only Send. Yes, even with small NFS READ payloads,
whose performance doesn't matter much anyway. Let's take that off
the table.

And I think we agree that payloads larger than a few dozen KB are
better transferred using explicit RDMA, for a variety of reasons.

IMO the only significant case where Send-based DDP is interesting
is replacing RDMA Read of moderate-sized payloads (either NFS
WRITE data, or Long Call messages). Let's look at NFS WRITE first.

Using a large inline threshold (say, 12KB), the server would have
to copy some or all of an 8KB payload into place after a large
Receive. Server implementations typically already have to copy
this data.

Why?

Because page-flipping a Receive buffer cannot work when the NFS
WRITE is modifying data in a file page that already has data on it.

Remember that Receive buffers are all the same, all RPC Call
traffic goes through them. The hardware picks the Receive buffer
arbitrarily; the ULP can't choose in advance which RPC goes into
which Receive buffer. Flipping the buffer is the only choice to
avoid a data copy.

Suppose the server platform's page size is 64KB, or even 2MB.
Then, 8KB NFS WRITEs are all sub-page modifications, and copying
would be required for each I/O. Explicit RDMA works fine in that
case without a data copy.

Host CPU data copy has comparable or better latency to flipping
a receive buffer into a file, allocating a fresh receive buffer,
and DMA mapping it, even for payloads as large as 4KB or 8KB.
Eliminating a server-side data copy on small NFS WRITEs is gravy,
not real improvement.

As far as I can tell, Send-based DDP is not able to handle the
sub-page case at all without a copy. That could add an extra rule
for using Send-based DDP over and above DDP eligibility, couldn't
it?

As soon as one realizes that Send-based DDP is effective only
when the payload is aligned to the server's page size, depends on
the surrounding file contents, and that it is less efficient than
explicit RDMA when the payload size is larger than a few KB,
introducing the complexity of Send-based DDP to the protocol or
an implementation becomes somewhat less appealing.

A better approach would attempt to address the actual problem,
which is the need to use RDMA Read for moderate payloads. This is
for both NFS WRITE and large RPC Call messages.

1. Long Call: large inline threshold or Message Continuation can
easily address this case; and it is actually not frequent enough
to be a significant performance issue.

2. NFS WRITE: Message Continuation works here, with care taken to
align the individual sub-messages to server pages, if that's
possible. Or, use push mode so the client uses RDMA Write in this
case.

The latter case would again be similar to DDP-eligibility, but not
the same. Same argument data items are allowed, but additional
conditions and treatment are required.

> I don't see how anything else matters.

It can matter greatly for NFS WRITE / RDMA Read.

In order for a server to set up the RDMA Read so the payload goes
into either a file's page cache or into the correct NVM pages,
the server has to parse the NFS header information that indicates
what filehandle is involved and the offset and count, before it
constructs the RDMA Read(s).

Without that information, the server performs the RDMA Read using
anonymous pages and either flips those pages into the file, or in
the case of NVM, it must copy the data into persistent storage
pages associated with that file.

That's the reason delayed RDMA Read is still important.

Now, Send-based DDP can handle NVM-backed files in one of two ways:

1. Receive into anonymous DRAM pages and copy the payload into NVM;
results in a (possibly undesirable) host CPU data copy

2. Receive into NVM Receive buffers; but then all of your NFS
operations will have to go through NVM. Maybe not an issue for
DRAM backed with flash, but it is a potential problem for
technologies like 3D XPoint which are 2x or more slower than
DRAM.

Remember, for RDMA Read, we are talking about NFS WRITE or
pulling moderate-sized arguments from the client on a Call. In
that case, a server-side host CPU copy is less likely to interfere
with an application workload, and thus is not as onerous as it
might be on the client.

> 	• There are no common cases in which it would provide a benefit.
> 	• It would complicate the design to make this something other than a drop-in replacement for V1-DDP>

Seems to me you can do a form of Send-based DDP with just
Message Continuation; perhaps just enough that an additional
Send-based DDP mechanism is not necessary.

And Send-based DDP isn't terribly valuable without a fully
send-based mechanism for transferring large payloads. So
it can't be done without Message Continuation, unless I'm
mistaken.

These two mechanisms seem well-entwined.

> Not sure what you mean  by that.

For Send/Receive, the receiver's host CPU always takes an
interrupt, and the host CPU performs transport header
processing. (even worse for Message Continuation, where one
RPC can now mean more than one interrupt on the receiving
CPU).

With RDMA Write, for example, the receiver does not take an
interrupt, and the wire header is processed by the HCA.

>  Both are the same in that:
> 	• You don't need to copy data on ether side
> 	• Both sides CPU"s need to deal with where the data is.  The adapter has to be told where the data is to sent/transferred from while the receiving side has to note where the placed data is (i.e not in the normal XDR stream).

Yes, it is. As above, sometimes with Receive a partial or
whole copy of the payload is unavoidable.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 3 Feb 2017 13:04:40 -0500

A lot of this reads like pieces of a review of rtrext.  I'll deal with some
issues within that later but first I want to focus on the concept of
DDP-eligibility.

I'm not sure exactly what it would mean for a concept such as this to be
completely general and I don't think that's the issue.  For me, the
important point is that it is a useful concept and that it has more than
one possible implementation.  XDR items are categorized as either
DDP-eligible or not.  I am not saying that additional useful
categorizations cannot arise in the future but I don't know of any and that
there doesn't seem to be any useful way to prepare for them.

The existing categorization is such that there will be individual instances
of items that implementations will not transfer using DDP.  This has always
been treated as an implementation choice and ULBs do not say, for example,
that READs of a few bytes are not DDP-eligible.  Instead, they are
DDP-eligible and requesters have a way to specify that they be returned
inline.  To do otherwise would make ULBs overly complicated and very hard
(perhaps Sisyphean) to maintain.  ULBs are limited to specifying which
items are DDP-eligible because that is what the requester and responder
need to agree on.  If the requester wants to send data inline or have
DDP-eligible response data returned inline that is up to him and ULBs do
not need to say anything about this choice.

The same applies when there are multiple possible forms of DDP with
different performance characteristics.  The requester chooses and the ULB
does not need to instruct him about which choice to make.  To try to create
rules about what choices the requester could make would make a ULB just
about impossible to do.  Let's not go there.

Now let me give some background about rtrext.  Perhaps I've been unduly
influenced by Talpey's figure of a million 8K random IOs per second, but
the fact is that I was focused on that case.  In my experience, it is a
very common case, deserving of significant attention.  I added message
continuation to rtrext to allow reads of multiples of the block size (e.g.
8K) to get the same sort of benefit.

Perhaps I did not devote the attention I should have to IO's that are not
an integral number of blocks but my experience is that clients are
typically structured to make this an unusual case.  YMMV.

I don't claim that send-based DDP is universally applicable and I expect
other forms of DDP to continue to exist and be useful.  With regard to the
READ/WRITE distinction, I accept that most of the benefits are for WRITE
and that many implentations might reasonably choose to use V1-like DDP for
READ while use send-based DDP for WRITE.  However, there are many
environments in which remote invalidation is not possible (e.g. user-level
server) and for these ther are significant benefits from using send-based
DDP for READ as well.

With regard to the sections that read like pieces of a review of rtrext, it
is hard to respond because the text often makes reference to potential
extensions for which there is no clear definition.   I think we have to
address these issues:

   - For push mode, it makes sense to define this as another optional
   extension to Version Two.  We need a volunteer to write rpcrdma-pushext.
   If we don't have one now, we should try again after rpcrdma-version-two
   becomes a working group document.
   - With regard to speculation about a possible form of form of Send-based
   DDP with just Message Continuation, I'm skeptical but if there were to be
   some clear definition, it could be added as an option to rtrext-0x, for x>=
   2.

With regard to issues related to challenges due to large page sizes, I see
the matter differently.  I think it is possible to avoid copying without
using page flipping,but using what I'll call "buffer flipping", even when
page sizes or large or there is no use of virtual memory at all.  One might
use page flipping only as a way to avoid copies on page-size units that are
smaller than the buffer size (e.g. 4K pages with 8K buffers).

Let's assume the buffer cache uses an 8K buffer size and that the page size
is either larger or non-existent (i.e.. no virtual memory).  Receive buffer
are structured to include a section for inline payload (1-2K) together with
an 8K buffer within the buffer cache.  When posted for receive, this buffer
should have no valid data within it and it the associated buffer header
should indicate that is neither free nor assigned to a particular file
location.  Once data is placed into it by a receive which gets a WRITE
request, that buffer can be assigned to a location within a file and the
buffer header updated to reflect this location.  The mapping from file and
block number to buffer address is changed but the address of the buffer
remains the same: it is the one into which the buffer was placed.

I think the conformance issues you have mentioned are worth thinking about
but there will be differences of opinion about how important these are.  In
any case, the discussion has resulted in some conclusions about use of
various form of DDP that need to be captured somewhere other than in an
email thread.

I can see why you don't want this stuff in rfc5667bis.  I don't think it
belongs in nfsulb (or in an update to rfc5667bis based on it) either.  For
now, I'll add an implementation-choice section to rtrext-02, but ultimately
I don't think this kind of stuff belongs in a standards-track document.  I
think that eventually, when the set of extensions in Version Two is more
settled (and larger than one), we could publish an Informational RFC that
provides implementation guidance about performance-relevant choices that
implementers might need to be made aware of.

:

On Thu, Feb 2, 2017 at 3:45 PM, Chuck Lever <[email-masked]> wrote:

> >       • Both sides CPU"s need to deal with where the data is.  The
> adapter has to be told where the data is to sent/transferred from while the
> receiving side has to note where the placed data is (i.e not in the normal
> XDR stream).
> > > >>> On Jan 27, 2017, at 6:40 PM, karen deitke <[email-masked]
> >
> > > >>>  wrote:
> > > >>>
> > > >>>
> > > >>> Hi Chuck,
> > > >>>
> protocol-
> > > >>> So you are saying for example that if the client sends a request
> in which it knows that it has "guessed" the maximum response but that guess
> > > >>> 4.5.  Why does the xid in the rdma header have to be used to get
> the session info, wouldnt the session info in the sequence_op that was sent
> in the first place have this info?
> > > >>>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 3 Feb 2017 16:09:55 -0500

> On Feb 3, 2017, at 1:04 PM, David Noveck <[email-masked]> wrote:
> 
> A lot of this reads like pieces of a review of rtrext.  I'll deal with some issues within that later but first I want to focus on the concept of DDP-eligibility.
> 
> I'm not sure exactly what it would mean for a concept such as this to be completely general and I don't think that's the issue.  For me, the important point is that it is a useful concept and that it has more than one possible implementation.  XDR items are categorized as either DDP-eligible or not.  I am not saying that additional useful categorizations cannot arise in the future but I don't know of any and that there doesn't seem to be any useful way to prepare for them.
> 
> The existing categorization is such that there will be individual instances of items that implementations will not transfer using DDP.  This has always been treated as an implementation choice and ULBs do not say, for example, that READs of a few bytes are not DDP-eligible.  Instead, they are DDP-eligible and requesters have a way to specify that they be returned inline.  To do otherwise would make ULBs overly complicated and very hard (perhaps Sisyphean) to maintain.  ULBs are limited to specifying which items are DDP-eligible because that is what the requester and responder need to agree on.  If the requester wants to send data inline or have DDP-eligible response data returned inline that is up to him and ULBs do not need to say anything about this choice.
> 
> The same applies when there are multiple possible forms of DDP with different performance characteristics.  The requester chooses and the ULB does not need to instruct him about which choice to make.  To try to create rules about what choices the requester could make would make a ULB just about impossible to do.  Let's not go there.
> 
> Now let me give some background about rtrext.  Perhaps I've been unduly influenced by Talpey's figure of a million 8K random IOs per second, but the fact is that I was focused on that case.  In my experience, it is a very common case, deserving of significant attention.  I added message continuation to rtrext to allow reads of multiples of the block size (e.g. 8K) to get the same sort of benefit.

There is no denying that is an important goal.

Unfortunately I've not found a way to approach that level
of throughput by shrinking per-op latency alone. It's
important to realize that there is a lower bound for per-op
latency that is determined by the physics of moving packets
on a physical network fabric.

High IOPS throughput is actually achieved by using multiple
QPs through multiple physical interfaces and networks linking
an SMB client and several storage targets (or in NFS terms:
pNFS).

So, if one RPC-over-RDMA connection can achieve 125KIOPS (and
I don't think that's unattainable, even with RPC-over-RDMA V1),
then eight connections might be able to swing a million IOPS,
given a careful client implementation. Getting to four
250KIOPS connections might be a challenge.

There are some critical pre-requisites to multi-channel
operation:

- bidirectional RPC on RPC-over-RDMA to enable NFSv4.1
- multi-path NFS (Andy's work), which works best with NFSv4.1
- pNFS/RDMA, which requires NFSv4.1, to enable the use of
multiple DSes

Linux is now close to having all of this in place.

> Perhaps I did not devote the attention I should have to IO's that are not an integral number of blocks but my experience is that clients are typically structured to make this an unusual case.  YMMV.

> I don't claim that send-based DDP is universally applicable and I expect other forms of DDP to continue to exist and be useful.  With regard to the READ/WRITE distinction, I accept that most of the benefits are for WRITE and that many implentations might reasonably choose to use V1-like DDP for READ while use send-based DDP for WRITE.  However, there are many environments in which remote invalidation is not possible (e.g. user-level server) and for these ther are significant benefits from using send-based DDP for READ as well.
> 
> With regard to the sections that read like pieces of a review of rtrext, it is hard to respond because the text often makes reference to potential extensions for which there is no clear definition.   I think we have to address these issues:
> 	• For push mode, it makes sense to define this as another optional extension to Version Two.  We need a volunteer to write rpcrdma-pushext.  If we don't have one now, we should try again after rpcrdma-version-two becomes a working group document.

At the moment, push mode looks like a new pNFS layout type,
thus it is independent of which version of RPC-over-RDMA is
in use. Agreed, that all has to be written up. Christoph or
I could make a start at it, but I'm not volunteering anyone's
time. It would be a short document similar to

http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.I-D.draft-hellwig-nfsv4-scsi-layout-nvme-00.xml

which unfortunately appears to have expired.

> 	• With regard to speculation about a possible form of form of Send-based DDP with just Message Continuation, I'm skeptical but if there were to be some clear definition, it could be added as an option to rtrext-0x, for x>= 2.

I was thinking of starting with an independent document
(and extension) just to draw the lines clearly. I've been
busy with other things, and haven't found an opportunity
to write it up.

> With regard to issues related to challenges due to large page sizes, I see the matter differently.  I think it is possible to avoid copying without using page flipping,but using what I'll call "buffer flipping", even when page sizes or large or there is no use of virtual memory at all.  One might use page flipping only as a way to avoid copies on page-size units that are smaller than the buffer size (e.g. 4K pages with 8K buffers). 
> 
> Let's assume the buffer cache uses an 8K buffer size and that the page size is either larger or non-existent (i.e.. no virtual memory).  Receive buffer are structured to include a section for inline payload (1-2K) together with an 8K buffer within the buffer cache.  When posted for receive, this buffer should have no valid data within it and it the associated buffer header should indicate that is neither free nor assigned to a particular file location.  Once data is placed into it by a receive which gets a WRITE request, that buffer can be assigned to a location within a file and the buffer header updated to reflect this location.  The mapping from file and block number to buffer address is changed but the address of the buffer remains the same: it is the one into which the buffer was placed.

In operating systems I'm familiar with, the page cache
is the single base of all file I/O. Sub-page sized I/O
(ie, a separate buffer cache) is no longer utilized.

Even if a particular file system implementation were
to use it, those buffers would not be exposed above
the VFS, and thus NFS servers would have no visibility
of them. Again, that's what I'm familiar with, maybe
not universally true, and perhaps I've misunderstood
something.

A Message Continuation approach could use a "flipping"
mechanism without the requester having to parcel out
chunk lists.

The key is separating the Transport header from the
Payload. By fixing the size of the Transport header,
a receiver can get the incoming Payload to land in
a separate buffer from the Transport header, always.

A sender might then rely on the fact that the
receiver's inline threshold is related to it's
preferred page/buffer cache size, and break each
message up so that certain data items in the Payload
stream land in independent buffers on the receiver
(again, this is similar to an experimental approach
being considered for NFS on TCP).

RDMA Receive preserves the sender's Send ordering.
The receiver simply concatenates the incoming messages
related to the same XID, grouping them as is convenient
to it.

Receiving such messages should work correctly for any
arbitrary division of Payload stream contents. Benefits
accrue when receivers can advertise to senders exactly
how they want to see RPC payloads divided into messages.
Perhaps a ULB could be useful here.

This arrangement is still unable to handle what I'll
term "zero copy sub-buffer modification". In more
abstract terms, I wonder if we can continue to label
this Direct Data Placement, since the buffer has to be
"flipped" into place after receive; it's not a bona
fide update-in-place. What we are attempting to do
here is more akin to zero-copy receive.

> I think the conformance issues you have mentioned are worth thinking about but there will be differences of opinion about how important these are.  In any case, the discussion has resulted in some conclusions about use of various form of DDP that need to be captured somewhere other than in an email thread.
> 
> I can see why you don't want this stuff in rfc5667bis.  I don't think it belongs in nfsulb (or in an update to rfc5667bis based on it) either.  For now, I'll add an implementation-choice section to rtrext-02, but ultimately I don't think this kind of stuff belongs in a standards-track document.  I think that eventually, when the set of extensions in Version Two is more settled (and larger than one), we could publish an Informational RFC that provides implementation guidance about performance-relevant choices that implementers might need to be made aware of.

Yes, that is one possibility.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sat, 4 Feb 2017 12:37:37 -0500

> I wonder if we can continue to label
> "flipped" into place after receive

Good observation :-), although somewhat annoying :-(.

In any case  rtrext-02 will refer to Send-based Data Placement.

On Fri, Feb 3, 2017 at 4:09 PM, Chuck Lever <[email-masked]> wrote:

> > A lot of this reads like pieces of a review of rtrext.  I'll deal with
> some issues within that later but first I want to focus on the concept of
> DDP-eligibility.
> >
> > The existing categorization is such that there will be individual
> instances of items that implementations will not transfer using DDP.  This
> has always been treated as an implementation choice and ULBs do not say,
> for example, that READs of a few bytes are not DDP-eligible.  Instead, they
> are DDP-eligible and requesters have a way to specify that they be returned
> > Perhaps I did not devote the attention I should have to IO's that are
> not an integral number of blocks but my experience is that clients are
> > With regard to the sections that read like pieces of a review of rtrext,
> it is hard to respond because the text often makes reference to potential
> >       • For push mode, it makes sense to define this as another optional
> http://xml2rfc.tools.ietf.org/public/rfc/bibxml3/reference.
> I-D.draft-hellwig-nfsv4-scsi-layout-nvme-00.xml
>
> >       • With regard to speculation about a possible form of form of
> Send-based DDP with just Message Continuation, I'm skeptical but if there
> were to be some clear definition, it could be added as an option to
> rtrext-0x, for x>= 2.
>
> > With regard to issues related to challenges due to large page sizes, I
> see the matter differently.  I think it is possible to avoid copying
> without using page flipping,but using what I'll call "buffer flipping",
> even when page sizes or large or there is no use of virtual memory at all.
> One might use page flipping only as a way to avoid copies on page-size
> units that are smaller than the buffer size (e.g. 4K pages with 8K buffers).
> >
> > Let's assume the buffer cache uses an 8K buffer size and that the page
> size is either larger or non-existent (i.e.. no virtual memory).  Receive
> buffer are structured to include a section for inline payload (1-2K)
> together with an 8K buffer within the buffer cache.  When posted for
> receive, this buffer should have no valid data within it and it the
> associated buffer header should indicate that is neither free nor assigned
> to a particular file location.  Once data is placed into it by a receive
> which gets a WRITE request, that buffer can be assigned to a location
> within a file and the buffer header updated to reflect this location.  The
> mapping from file and block number to buffer address is changed but the
> address of the buffer remains the same: it is the one into which the buffer
> was placed.
>
> > I think the conformance issues you have mentioned are worth thinking
> about but there will be differences of opinion about how important these
> are.  In any case, the discussion has resulted in some conclusions about
> use of various form of DDP that need to be captured somewhere other than in
> an email thread.
> >
> > > > >>> On Jan 27, 2017, at 6:40 PM, karen deitke <
> [email-masked]>
> > > > >>>  wrote:
> > > > >>>
> > > > >>>
> > > > >>> Hi Chuck,
> > > > >>>