Subject: [nfsv4] I-D Action: draft-ietf-nfsv4-rfc5667bis-04.txt

From: [email-masked]
Date: Fri, 20 Jan 2017 16:27:20 -0800

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Network File System Version 4 of the IETF.

        Title           : Network File System (NFS) Upper Layer Binding To RPC-Over-RDMA
        Author          : Charles Lever
	Filename        : draft-ietf-nfsv4-rfc5667bis-04.txt
	Pages           : 18
	Date            : 2017-01-20

Abstract:
   This document specifies Upper Layer Bindings of Network File System
   (NFS) protocol versions to RPC-over-RDMA.  Upper Layer Bindings are
   required to enable RPC-based protocols, such as NFS, to use Direct
   Data Placement on RPC-over-RDMA.  This document obsoletes RFC 5667.

The IETF datatracker status page for this draft is:
https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5667bis/

There's also a htmlized version available at:
https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5667bis-04

A diff from the previous version is available at:
https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5667bis-04

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

---

From: karen deitke <[email-masked]>
Date: Fri, 27 Jan 2017 16:40:45 -0700

Hi Chuck,

o Section 2.6, second to last paragraph, typo, priori should be prior.
o Same section, "Note however that many operations normally considered 
non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".  This was 
confusing because prior in the paragraph you are talking about determine 
the reply size for variable length replies, which doesn't seem to be the 
case for WRITE, SETATTR.
o 4.2.1 seoond paragrah would read better if you removed the word "however"
"If a client implementation is equipped to recognize that a transport 
error could mean that it provisioned an inadequately sized Reply chunk, 
it can retry the operation with a larger Reply chunk. Otherwise, the 
client must terminate the RPC transaction."

So you are saying for example that if the client sends a request in 
which it knows that it has "guessed" the maximum response but that guess 
may be wrong, if it detects the connection dropped for example, it could 
potentially interpret that as the guess was too small and send again?

4.3, second from last bullet, "If an READ", should be "If a READ"

4.5.  Why does the xid in the rdma header have to be used to get the 
session info, wouldnt the session info in the sequence_op that was sent 
in the first place have this info?

Karen
On 1/20/2017 5:27 PM, [email-masked] wrote:
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: David Noveck <[email-masked]>
Date: Sat, 28 Jan 2017 06:54:49 -0500

> So you are saying for example that if the client sends a request in which
it knows that it has "guessed"
> the maximum response but that guess may be wrong, if it detects the
connection dropped for example,
> it could potentially interpret that as the guess was too small and send
again?

That's how I interpreted that sentence, but I don't think responders should
be doing things that
cause the connection to be dropped, and I think that 5667bis should be
giving some sort of
guidance about how to avoid such situations

We are supposing that requester C guesses the that reply to request R can
be no longer than length
L1 and that the responder S determines that the reply length is L2, with L2
> L1.  The responder knows
L1 and L2 and will be aware that the reply chink is too small.  It seems
that you (and possibly Chuck
as well) are assuming the responder/server is gong to ignore this mismatch
and do something that
causes an operational error as described in rfc666bis.  I don;t see why a
server would do that and I
think that it is OK for the ULB to tell the responder that he SHOULD NOT
(or even MUST NOT)
do that.

Regarding the error to be returned, for Version One, I think ERR_CHUNK
would fill the bill even though
this can also be returned in the event of an XDR error RPC-over-RDMA
request header.   Future
versions should have an explicIt error code for this case.  Until this
done, we are not ready for
draft-ietf-nfsv4-rpcrdma-version-two to be submitted.

On Fri, Jan 27, 2017 at 6:40 PM, karen deitke <[email-masked]>
wrote:

> "If a client implementation is equipped to recognize that a transport error
> could mean that it provisioned an inadequately sized Reply chunk, it can
> retry the operation with a larger Reply chunk.  Otherwise, the client
> must terminate the RPC transaction."
>
> So you are saying for example that if the client sends a request in which
> it knows that it has "guessed" the maximum response but that guess may be
> wrong, if it detects the connection dropped for example, it could
> session info, wouldnt the session info in the sequence_op that was sent in
> the first place have this info?
>
> Karen
>
> The IETF datatracker status page for this draft is:https://datatracker.ietf.org/doc/draft-ietf-nfsv4-rfc5667bis/
>
> There's also a htmlized version available at:https://tools.ietf.org/html/draft-ietf-nfsv4-rfc5667bis-04
>
> A diff from the previous version is available at:https://www.ietf.org/rfcdiff?url2=draft-ietf-nfsv4-rfc5667bis-04
>
>
> Internet-Drafts are also available by anonymous FTP at:ftp://ftp.ietf.org/internet-drafts/
>
> nfsv4 mailing [email-masked]://www.ietf.org/mailman/listinfo/nfsv4
>
>
>

---

From: Chuck Lever <[email-masked]>
Date: Sat, 28 Jan 2017 12:16:43 -0500

> On Jan 28, 2017, at 6:54 AM, David Noveck <[email-masked]> wrote:
> 
> > So you are saying for example that if the client sends a request in which it knows that it has "guessed" 
> > the maximum response but that guess may be wrong, if it detects the connection dropped for example, 
> > it could potentially interpret that as the guess was too small and send again?

"Transport error" here means an RDMA_ERROR, not necessarily a
dropped connection.

> That's how I interpreted that sentence, but I don't think responders should be doing things that
> cause the connection to be dropped, and I think that 5667bis should be giving some sort of

As this is a common issue for ULPs, a ULB's authors might rightly
assume that rfc5666bis already provides reasonable guidance about
it, thus it wouldn't have to be covered again in each ULB, unless
that ULP requires finer-grained mechanisms for reporting a lack
of reply resources. I don't believe that NFS has such requirements.

> We are supposing that requester C guesses the that reply to request R can be no longer than length
> L1 and that the responder S determines that the reply length is L2, with L2 > L1.  The responder knows
> L1 and L2 and will be aware that the reply chink is too small.  It seems that you (and possibly Chuck
> as well) are assuming the responder/server is gong to ignore this mismatch and do something that
> causes an operational error as described in rfc666bis.

IMO a responder that drops the connection in this case is an
immature implementation. A responder can usually tell when a
Reply chunk is too small before it posts an RDMA Write to fill
the chunk, and that is a reasonable sanity check to perform
before starting to construct and send a reply.

> I don;t see why a server would do that and I
> think that it is OK for the ULB to tell the responder that he SHOULD NOT (or even MUST NOT)
> do that.

Again, rfc5666bis (or rpcrdma-version-two) is a better place for
this kind of mandate, IMO, unless a ULP has specific additional
requirements.

In other words, the reason a ULB would prescribe particular
responses to this situation would be that the existing text in
rfc5666bis is not adequate or suited to the ULP in question.

> Regarding the error to be returned, for Version One, I think ERR_CHUNK would fill the bill even though
> this can also be returned in the event of an XDR error RPC-over-RDMA request header.

rfc5666bis Section 4.5.3 says that the responder should send
ERR_CHUNK. However, Section 4.5 and 4.5.3 state that RDMA_ERROR
indicates a permanent error (in other words, a V1 requester
should not retry the operation).

Note that the idea of retrying is recent, and not part of
RPC-over-RDMA Version One. The language in Section 4.2.1 of
rfc5667bis is there because some future version of RPC-over-RDMA
might provide a proper mechanism for recovering from this
situation.

However, I find it challenging to compose a ULB that can
accommodate non-existent future transports, so I wonder if we
would be better served by trying not to predict the future in
ULB specifications. Leaving this kind of language to the
transport specifications seems like a stronger architecture.

> Future 
> versions should have an explicIt error code for this case.  Until this done, we are not ready for 

Indeed, rpcrdma-version-two-03 Section 6.2.4 introduces
RDMA2_ERR_REPLY_RESOURCE for this purpose.

> On Fri, Jan 27, 2017 at 6:40 PM, karen deitke <[email-masked]> wrote:
> 
> Hi Chuck,
> 
> o Same section, "Note however that many operations normally considered non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".  This was confusing because prior in the paragraph you are talking about determine the reply size for variable length replies, which doesn't seem to be the case for WRITE, SETATTR.
> "If a client implementation is equipped to recognize that a transport error could mean that it provisioned an inadequately sized Reply chunk, it can retry the operation with a larger Reply chunk.  Otherwise, the client must terminate the RPC transaction."
> 
> So you are saying for example that if the client sends a request in which it knows that it has "guessed" the maximum response but that guess may be wrong, if it detects the connection dropped for example, it could potentially interpret that as the guess was too small and send again?
> 
> 4.5.  Why does the xid in the rdma header have to be used to get the session info, wouldnt the session info in the sequence_op that was sent in the first place have this info?
> 
> Karen
> 
>> until the htmlized version and diff are available at 
>> tools.ietf.org
>> .
>> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Sat, 28 Jan 2017 13:20:24 -0500

> of reply resources. I don't believe that NFS has such requirements.

That makes sense but:

   - There is only a single ULB so in trying to address this in the
   transport spec, we are tring to future-proof the spec, which is desirable
   to do, if we don;t thereby ttie ourselves in knots.
   - In the case of Version One, it isn't clear exactly what we will say
   about this in rfc5666bis.

It should noted  that this prediction problem only applies to a subset of
the ULP's
discussed in rfc5667: NFSv4.0 and some of the auxiliary prtocols use
together with
v2 and v3.

> immature implementation.

I agree but, given that we are writing standards track documents, we have
to decide whether to define
such implementation as broken.  In other words, is the client allowed to
rely on the server not being
immature in this way?

> transport specifications seems like a stronger architecture.

I'm not sure exactly what text you are proposing to put into rfc5667bis to
deal with
(or not deal with) this issue.

My preference would be for rfc5667bis to adapt to your stronger
architecture by
saying:

   - It is the responsibility of the transport version to define how the
   responder is to deal with the case of incorrect reply size prediction.
   - This definition might include the specific errors to use to report
   this issue and also might specify that this is a permanent error, for which
   retry is not appropriate. (although I would hope that this could be avoided
   in the case of Version One).
   - Specifying that  when the transport indicates the problem and allows
   the retry, the ULP's which have the issue should retry.
   - Indicates which ULPs discussed within this ULB have the prediction
   issue.

I don't think we can simply say nothing about the issue in rfc5667bis,
since, if we did that, we would
need an rfc5667bisbis as part of doing Version Two.  I'd prefer if we
didn't have to do that.

> RDMA2_ERR_REPLY_RESOURCE for this purpose.

Great, so this is not a bar to submission of draft-ietf-nfsv4-rpcrdma-
version-two.
There are probably other things we have to do to allow that to be done,but
that is a
discussion it is best to have later.

On Sat, Jan 28, 2017 at 12:16 PM, Chuck Lever <[email-masked]>
wrote:

> which it knows that it has "guessed"
> request header.
> error could mean that it provisioned an inadequately sized Reply chunk, it
> can retry the operation with a larger Reply chunk.  Otherwise, the client

---

From: Chuck Lever <[email-masked]>
Date: Mon, 30 Jan 2017 11:05:12 -0500

> On Jan 28, 2017, at 1:20 PM, David Noveck <[email-masked]> wrote:
> 
> 	• There is only a single ULB so in trying to address this in the transport spec, we are tring to future-proof the spec, which is desirable to do, if we don;t thereby ttie ourselves in knots.
> 	• In the case of Version One, it isn't clear exactly what we will say about this in rfc5666bis.
> It should noted  that this prediction problem only applies to a subset of the ULP's 
> discussed in rfc5667: NFSv4.0 and some of the auxiliary prtocols use together with 
> v2 and v3.
> 
> I agree but, given that we are writing standards track documents, we have to decide whether to define 
> such implementation as broken.  In other words, is the client allowed to rely on the server not being
> immature in this way?

In my view, there are special two cases rfc5667bis has to deal with on
behalf of NFS:

1. How to handle non-idempotent ULP operations in the face of potential
transport layer retries. This is a problem for all the ULPs in rfc5667bis
except NFS versions that have a session layer. Is this also a problem
for RPC-over-TCP?

2. NFSv4+ requires that RPCs cannot be retransmitted on the same
connection, and NFSv4 clients depend on servers to disconnect when they
have dropped a request. Does ERR_CHUNK count as a dropped request? Should
clients retransmit if the server drops a connection because it cannot
reply? This issue has to be addressed for all NFSv4 minor versions.

In other words, a disconnect already has a particular meaning to NFSv4:
please retransmit. V1's ERR_CHUNK has the opposite meaning: please do not
retransmit.

A responder may disconnect at any time. Requesters have to be prepared
for that, and I don't see how we can say "a responder may disconnect any
time except when a requester has provided inadequate reply resources."

So, I might propose something for rfc5667bis along the lines of:

- If there is a disconnect and the responder has provided no other
response for an XID, the ULP rules for handling retransmission apply.

- If there is a transport error indicated (ie, RDMA_ERROR) before
the disconnect, then the requester must respond to that error as
prescribed by the transport specification.

> I'm not sure exactly what text you are proposing to put into rfc5667bis to deal with
> My preference would be for rfc5667bis to adapt to your stronger architecture by
> saying:
> 	• It is the responsibility of the transport version to define how the responder is to deal with the case of incorrect reply size prediction.
> 	• This definition might include the specific errors to use to report this issue and also might specify that this is a permanent error, for which retry is not appropriate. (although I would hope that this could be avoided in the case of Version One).
> 	• Specifying that  when the transport indicates the problem and allows the retry, the ULP's which have the issue should retry.
> 	• Indicates which ULPs discussed within this ULB have the prediction issue.
> I don't think we can simply say nothing about the issue in rfc5667bis, since, if we did that, we would
> need an rfc5667bisbis as part of doing Version Two.  I'd prefer if we didn't have to do that.
> 
> 
> > > Future
> Great, so this is not a bar to submission of draft-ietf-nfsv4-rpcrdma-version-two.
> There are probably other things we have to do to allow that to be done,but that is a 
> On Sat, Jan 28, 2017 at 12:16 PM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Mon, 30 Jan 2017 11:58:59 -0500

> except NFS versions that have a session layer. Is this also a problem

No and the reason is instructive.  In the case of TCP, a transport layer
retry, e,g, because a frame was dropped, one is sure that the server
never saw the original request.  The same for retries done by IB
adapters.

In the case of RPC-over-RDMA, we seem to be talking about a
retry due to the fact that that the reply chunk was inadequate as
a transport layer retry.  Is that the kind of retry you were referring to?

> have dropped a request. Does ERR_CHUNK count as a dropped request?

I don't think it should.  The reason for this requirement was to prevent
an accumulation copies of the same request.  In the case, of ERR_CHUNK,
one has the option of changing the request (e.g. by using a bigger
reply chunk) and re-issuing, so that forcing a reconnection is irrelevant.
In other words, I think it is OK to retry with a bigger reply chunk but not
OK
so simply resend the same request.  That's BOGUS (to be defined in an
rfc2219bis that I'll submit on 4/1).

> Should

Clients cannot know why the server drops a connection.  Also, I
don't see why he should drop the connection (as opposed to
sending ERR_CHUNK), simply because the client has sent an
inadequate reply chunk.  You can call it "BOGUS" or "immature"
as you wish, but we should not be assuming that this is a reasonable
server response to a client screwup.

BTW, section 2.6 of rfc566bis-04, in explaining why reliable reply size
estimation is necessary seems to imply it is a possible response:

A Reply buffer overrun can result in corruption of the Reply message
or termination of the transport connection.

I think is this left over from when ERR_CHUNK was not used to signal an
inadequate reply chunk.  sigh!.

> please retransmit.

I think the meaning is more like "you can retransmit or fail the request as
you choose".

> V1's ERR_CHUNK has the opposite meaning: please do not retransmit.

I would say the meaning is "if you retransmit without changing
anything, the same things is going to happen".  I think retransmission after
addressing the likely problem, should be allowed.

> time except when a requester has provided inadequate reply resources."

I don't either

> response for an XID, the ULP rules for handling retransmission apply.

OK.

> prescribed by the transport specification.

This has the unfortunate consequence of not allowing retry with a bigger
reply chunk even when idempotence.is not an issue.  However, I can
live with it, siince it allows he problem to be fixed cleanly in V2.

On Mon, Jan 30, 2017 at 11:05 AM, Chuck Lever <[email-masked]>
wrote:

> >       • There is only a single ULB so in trying to address this in the
> >       • It is the responsibility of the transport version to define how
> the responder is to deal with the case of incorrect reply size prediction.
> >       • This definition might include the specific errors to use to
> report this issue and also might specify that this is a permanent error,
> for which retry is not appropriate. (although I would hope that this could
> be avoided in the case of Version One).
> >       • Specifying that  when the transport indicates the problem and
> allows the retry, the ULP's which have the issue should retry.

---

From: karen deitke <[email-masked]>
Date: Mon, 30 Jan 2017 10:01:34 -0700

> prescribed by the transport specification.
This is kind of a side note, but it would be nice if RDMA_ERROR, 
ERR_CHUNK could indicate which chunk was in error, read/write or reply.

Karen

---

From: David Noveck <[email-masked]>
Date: Mon, 30 Jan 2017 15:32:24 -0500

It can't be addressed in Version One, as the XDR is considered fixed.

Version Two already has a lot of what you are looking for and the errors
are open to further refinement.  Look at
draft-cel-nfsv4-rpcrdma-version-two-03 and look at rpcrdma2_errcode which
already has six distinct error codes.  I think more can be added.  Also the
union rpcrdma2_error allows specific info to be added with error details
for any particular code. Please propose the changes you think are needed.

In any case, more review and comment would be helpful to get this ready to
be converted into a working group document.

BTW, it seems that there is nothing in the Version Two document about
reporting eligibility violations.  I think we need a specific code for that
in Version Two.

On Mon, Jan 30, 2017 at 12:01 PM, karen deitke <[email-masked]>
wrote:

> This is kind of a side note, but it would be nice if RDMA_ERROR, ERR_CHUNK
> could indicate which chunk was in error, read/write or reply.
>
> Karen
>
>
>>
>>>         • It is the responsibility of the transport version to define
>>> how the responder is to deal with the case of incorrect reply size
>>> prediction.

---

From: Chuck Lever <[email-masked]>
Date: Mon, 30 Jan 2017 19:28:13 -0500

Hi Karen, thanks for taking time to review.

> On Jan 27, 2017, at 6:40 PM, karen deitke <[email-masked]> wrote:
> 
> 
> Hi Chuck,
> 

"a priori" is Latin for "in advance" (literally, "from the earlier"). I've
replaced this phrase.

> o Same section, "Note however that many operations normally considered non-idempotent (e.g. WRITE, SETATTR) are actually idempotent".  This was confusing because prior in the paragraph you     are…

That's correct, WRITE doesn't have a variable length reply. This section
discusses a possible COMPOUND with WRITE (fixed length result, non-idempotent)
and GETATTR (variable length result, idempotent). I've clarified it.

> o 4.2.1 seoond paragrah would read better if you removed the word "however"

I was not able to find the word "however" in the second paragraph of S4.2.1.

> So you are saying for example that if the client sends a request in which it knows that it has "guessed" the maximum response but that guess may be wrong, if it detects the connection dropped for e…

As previously discussed.

> 4.3, second from last bullet, "If an READ", should be "If a READ"

Fixed.

> 4.5.  Why does the xid in the rdma header have to be used to get the session info, wouldnt the session info in the sequence_op that was sent in the first place have this info?

Here's the last paragraph of Section 4.5:

   In addition, within the error response, the requester does not have
   the result of the execution of the SEQUENCE operation, which
   identifies the session, slot, and sequence id for the request which
   has failed.  The xid associated with the request, obtained from the
   rdma_xid field of the RDMA_ERROR or RDMA_MSG message, must be used to
   determine the session and slot for the request which failed, and the
   slot must be properly retired.  If this is not done, the slot could
   be rendered permanently unavailable.

The mention of RDMA_MSG is probably incorrect. In that case, the server
was able to return a SEQUENCE result, and that should be useable by the
client.

Without an RPC Reply message, however, the client matches the XID in the
ERR_CHUNK message to a previous call and that will have the matching
SEQUENCE operation.

That makes this rather a layering violation, and perhaps a reason why
retransmitting with a larger Reply chunk might be a cure worse than the
disease.

I'm beginning to believe that making this situation always a permanent
error, as rfc5666bis does, is a better protocol choice.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 31 Jan 2017 07:11:41 -0500

> ERR_CHUNK message to a previous call and that will have the matching

Right.

> That makes this rather a layering violation,

It looks to me like this layering violation is not serious and I think it is
unavoidable.

> and perhaps a reason why

Note that the layering violation does not arise when retrying the operation
with a larger reply chunk.

The case when this layering violation/misdemeanor occurs is when
the operation is failed and the slot needs to be made available again.
Retrying
the operation with a larger reply chunk would make make this situation.
less likely to occur. However, since 4.1 has a reliable means of limiting
reply
size (unlike v4.0), it appears that this is beside the point for
session-based
minor versions of nfsv4.

Note that the disease here is the fact that v4.0  (and some auxiliary
protocols) does not have a reliable means of determining reply size limits.
We can't cure that disease as these protocols are unchangeable.

Allowing retry with a larger reply chunk is not a cure but it is a
treatment which
ameliorates the problem.  As far as I can see, it is a safe and effective
treatment.

> error, as rfc5666bis does, is a better protocol choice.

I don;t see it that way.  It leaves us with an rfc5666bis  requirement for
ULBs that
we would be unable to satisfy for a number of ULPs dealt with in rfc5667bis.

On Mon, Jan 30, 2017 at 7:28 PM, Chuck Lever <[email-masked]> wrote:

> confusing because prior in the paragraph you     are talking about
> determine the reply size for variable length replies, which doesn't seem to
> be the case for WRITE, SETATTR.
>

---

From: Chuck Lever <[email-masked]>
Date: Tue, 31 Jan 2017 12:04:14 -0500

> On Jan 31, 2017, at 7:11 AM, David Noveck <[email-masked]> wrote:
> 

As I clicked Send yesterday, I blinked and "retried" became "retired".
Although I'm a fan of "retired", a different term might be less likely
to confuse readers in this context. Suggestions welcome.

So now I read this text

>    be rendered permanently unavailable.

to mean "No short Reply chunk retries are permitted when a session is
in use." I'm much more comfortable with this requirement.

And, now there is one less use case for retrying a short Reply chunk.

> It looks to me like this layering violation is not serious and I think it is

Let's consider how the layers might work together in this case.

Sizing a Reply chunk is done by the ULP. If the Reply chunk turns out
to be too small, it is the ULP, not the transport, that will have to
replace that chunk with a larger Reply chunk.

The ULP therefore must be fully aware that the previous attempt
failed, and why. It also has to be capable of deciding whether or not
a retry is advisable, given other factors such as responder reply
caching.

Since the ULP is driving the retry, given the separation of duties
between a ULP and the RPC layer, it is possible that a new XID will
be used for the retried operation. The responder does not have a way
of recognizing the link between the failed XID and the new one.

To address this, then the ULP would have to indicate what XID should
be used in the retry's RPC (and RPC-over-RDMA) header.

> the operation is failed and the slot needs to be made available again.  Retrying
> less likely to occur. However, since 4.1 has a reliable means of limiting reply 
> size (unlike v4.0), it appears that this is beside the point for session-based
> minor versions of nfsv4.
> 
> Allowing retry with a larger reply chunk is not a cure but it is a treatment which 
> ameliorates the problem.  As far as I can see, it is a safe and effective treatment.

The safe and most conservative course is to terminate the RPC, in all
cases. Some ULPs might be able to tolerate retry, others might not.

Each ULP has to make that decision on a case-by-case basis. The
transport by itself cannot arbitrarily perform a retry, it is now
clear.

Thus I'm considering removing the retry language from rfc5667bis.

First, the mandate I recall receiving in Dallas for rfc5667bis was:

1. Document existing implementations
2. Fix mistakes and omissions
3. Extend the NFS ULB to properly cover NFSv4.1 and NFSv4.2

Since then we have added:

4. Make it align with new language in rfc5666bis

And snuck in:

5. Make it align with the language of rpcrdma-version-two and
rpcrdma-rtrext (which I'm also struggling with, but that's a
separate topic)

As I see it, "retrying short Reply chunks" is a new and untried
piece of protocol. It attempts to accommodate a future,
unimplemented transport or ULP, and therefore it lies outside the
mandated scope of rfc5667bis IMO.

Second, it's unnecessary in the real world:

I'm not aware of a single instance in the field where a server has
had to reject a client's request because of a short Reply chunk.
It's certainly never occurred during my testing, but lots of other
issues have.

Third, the places where retry might be useful are all legacy
protocols:

As you pointed out, NFSv4.0 and NFSACL appear to be the two areas
where we have concerns. The problem is largely addressed by NFSv4.1
and newer minor versions, and even there, retrying is not permitted.
The future does not need this new behavior, apparently.

What retrying amounts to, in this context, is adding a workaround
in the transport protocol for bugs in the Upper Layer Protocols and
their implementations.

> I don;t see it that way.  It leaves us with an rfc5666bis  requirement for ULBs that 

This is a hole that can be closed simply by prescribing how each
implementation must behave when a Reply chunk is short. That has
already been done in rfc5666bis: the RPC fails. This prevents a
transport deadlock and connection loss, and indicates a ULP
implementation bug.

Why does rfc5667bis need to go further? The consequences of a
shortage are clear, and no longer catastrophic to other RPCs.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 31 Jan 2017 14:54:33 -0500

> Thus I'm considering removing the retry language from rfc5667bis.

I don't think that should be done.  Let me explain why.

> 3. Extend the NFS ULB to properly cover NFSv4.1 and NFSv4.2

And those have been done :-)

> 4. Make it align with new language in rfc5666bis

One part of which is to require that you have a reliable means of
bounding reply sizes.  Allowing retry is a means to accomplish
that requirement without essentially waving the problem away
and guessing/hoping that it will not be bothersome.   Even if this
is not a problem in practical terms, I don;t see how we can present
an rfc5667bis that doesn't meet the requirements in rfc5666bis.

> And snuck in:

I don't think there was anything sneaky here.  The "mandate"
you mention was not a legislative act or executive order.
It was a plan to go forward with and we should be able
to make changes if necessary.

> rpcrdma-rtrext (which I'm also struggling with, but that's a

I think the goal is not to align it with the particulars of any
particular version or extension but to make it generic, so
that the transport can be changed without continually revising
the ULB for new versions and extensions.  The concept of an
extensible version 2 transport would become unworkable in that
case.

As an example, the current rfc5667bis, in numerous places
assumes that no mesaage continuation feature is available.  We have
suffered, for a long time, from the lack of this feature which SMB
Direct has had for a long time.  Even apart from the controversial
concept of send-based DDP, I don't see how we can have an rfc5667bis
that essentially forecloses a message continuation extension.

> mandated scope of rfc5667bis IMO.

But that position would essentially ties us to Version One forever, which,
aside from the particulars of the discussion in Dallas, is not where we want
to be.,

I think Version One can accommodate retry, even though it
has some weaknesses with regard to non-idempotent operations.
It is true that version Two will probably have better, more complete
support.

It may be that there is something in rfc566bis as currently written, that
makes this difficult to do at this late stage.  I could live with that,
but given that rfc5667bis needs to be made transport-generic, I think
it should definitely allow this in future transport Versions, if those
versions support it, and not forclose it, just as it should not foreclose
message continuation.

> Second, it's unnecessary in the real world:

I agree that it is very unlikely to be used and that many implementations
may choose not to implement it.

> It's certainly never occurred during my testing, but lots of other

OK, but the problem is that, without it, when and if this happens,
we have a situation in which it is clear there is a bug, but it cannot be
determined whether the fault is due to the requester or the responder.

In order to allow interoperable implementations, this kind of situation
must be avoided.

If the spec allows retry but the requester implementation does not provide
it,
then it has a bug and it can decode it is willing to live with a small
possibility of
failure.

>Third, the places where retry might be useful are all legacy

If you use the term "legacy protocols" as it is used in rfc5667bis, this is
not so.
Also, rfc5666bis requires reliable reply estimation of all ULPs, rather
than
only of non-legacy ones.

---

From: David Noveck <[email-masked]>
Date: Tue, 31 Jan 2017 15:33:06 -0500

> It looks to me like this layering violation is not serious and I think it
is

Note that that the (not very serious) layering violation occurs when the
request is failed independent of whether retry is attempted or not.

> Let's consider how the layers might work together in this case.

The way you propose (not quoted) doesn't seem to work very well
but let me outline one that I think works better:

Sizing a Reply chunk is done by the ULP which proposes a size and an
indication
of whether that size is reliable (not to be retried) or a best guess (open
to retry by
the transport, if it has retry support).  If the Reply chunk turns out to
be too small,
and the ULP indicates it is a best guess, the transport, will replace the
reply chunk
 by a larger one (e.g. by doubling it).  It continues this until it
succeeds or is unable to
keep raising the size, at which point it fails the request.  Dealing with
the failure
case involves the abovementioned layering misdemeanor but the retry  case
doesn't.

> implementation must behave when a Reply chunk is short.

I don't see how that meets the ULB requirement of rfc5666bis.

> That has
> transport deadlock and connection loss,

All desirable things, but ...

> and indicates a ULP implementation bug.

But it is not clear whether the ULP implementation bug is in the client
or the server.  When this happens, the server implementers are usually
certain it is the client's fault and vice versa.   That makes it hard to get
the problem fixed, which is one reason standards try to be clear about
defining correct behavior and thus who is at fault.

> Why does rfc5667bis need to go further?

I think you signed up to do this in rfc566bis.  I think it would be a lot
of trouble to modify this
requirement now,

> The consequences of a

Avoiding catastrophe, while a good and necessary thing, is not sufficient.

---

From: Chuck Lever <[email-masked]>
Date: Tue, 31 Jan 2017 18:02:53 -0500

> On Jan 31, 2017, at 2:54 PM, David Noveck <[email-masked]> wrote:
> 

> the ULB for new versions and extensions.

I've attempted to make the language more generic, but I
regard that as a constructive but ultimately unattainable
goal because it requires predicting the future with a high
degree of accuracy.

At some point we have to stop and say it is good enough.

> The concept of an

IMO that's overstating it. We have a lot to learn about how to
extend RPC-over-RDMA. The extension rules in rpcrdma-version-two
have to state how extension specifications deal with adding
new ULB requirements. Right now, we don't know what requirements
even existing proposed extensions might add. For instance:

- Do we really believe that rfc5666bis DDP eligibility is an
appropriate rule to apply, without any change, to send-based DDP?
I don't think that's a foregone conclusion.

- Can you explain here why you believe that message continuation
will have an impact on the way ULBs are written?

The point being that we have a set of ULB composition rules in
rfc5666bis. You seem to be adding new ones that we haven't all
agreed on, and then dinging me (repeatedly) for ignoring them.

> that essentially forecloses a message continuation extension.

I'm much more comfortable closing the book on rfc5667bis with the
set of transports we have in RFCs now, and then opening another
update when we have real documented transport changes to address.
Otherwise rfc5667bis will never be finished.

Let's design for what we have, clear the table, and move on. There's
no reason to believe that will create an unending sequence of updates
to the NFS ULB. There will be no more ULB updates than there are
extensions, and quite likely fewer.

> aside from the particulars of the discussion in Dallas, is not where we want

It doesn't tie anyone to anything. The WG is allowed to open
updates of existing RFCs as many times as they like. I believe
there is adequate process here to deal properly with it.

> has some weaknesses with regard to non-idempotent operations.

ERR_CHUNK can be returned for a lot of reasons. As we learned
when trying to add extensibility to V1, that could make it
inadequate for this purpose.

If short Reply chunk retry is mentioned in rfc5667bis, I prefer
that it is discussed in the following terms:

1. It is an optional remedy, written as implementation advice.

2. The choice to retry is at the discretion of the ULP on the
requester (not the transport)

3. Retry is done via a distinct RPC (fresh XID)

4. Short Reply chunk retry is permitted only for NFSACL GETACL
and NFSv4.0 GETATTR

> It is true that version Two will probably have better, more complete 

Possibly all that is necessary is an unambiguos error code.

> versions support it, and not forclose it, just as it should not foreclose 

Another way to "allow retry in future versions" is not to
mention it at all, which is what I proposed in the previous
e-mail.

> determined whether the fault is due to the requester or the responder.

The responder reports, via a local system log, that the requester
is in error. The Solaris NFS/RDMA server implementation reports
similar errors this way all the time, in addition to sending
GARBAGE_ARGS replies, so I know this works.

It can be done with V1 implementations, and it can be done even
if requesters do not support retrying.

> If the spec allows retry but the requester implementation does not provide it, 
> then it has a bug and it can decode it is willing to live with a small possibility of 
> failure. 
> 
> If you use the term "legacy protocols" as it is used in rfc5667bis, this is not so.

To be clear, I meant legacy in the sense that the book is closed
on NFSACL and NFSv4.0.

> Also, rfc5666bis requires reliable reply estimation of all ULPs, rather than 
> only of non-legacy ones.

The intent of that requirement is to ensure interoperation.

You haven't explained why "failing a request that is not formed
according to the rules" is not sufficient to establish that
implementations interoperate successfully.

If the rfc5666bis requirement is too stern or hazy or in some
other way not sensible, we should fix it.

But I don't see anything that states a ULB is prevented from
identifying gray areas or possible cases that can't be handled.

> On Tue, Jan 31, 2017 at 12:04 PM, Chuck Lever <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 31 Jan 2017 19:59:58 -0500

> goal because it requires predicting the future with a high

I don't agree.  I think we need use more abstraction in
stating the requirements than rfc5667 did.  The NFS specs
work with filesystems that were not known when the
spec were written.  Only when we extend the
abstraction, as we did with xattrs, do we have to
write a new extension document.

> At some point we have to stop and say it is good enough.

We agree on that but disagree about what that point is.

> IMO that's overstating it. We have a lot to learn about how to
extend RPC-over-RDMA.

> The extension rules in rpcrdma-version-two
> new ULB requirements.

I don;t think extensions need to add ULB requirements.  I do think we
need some text in rpcrdma-version-two that explains what the extensions
have to say if they propose different implementation of abstractions
referred to by ULB(s).

I'll draft some proposed text in this regard and run it by you.

> Right now, we don't know what requirements

> appropriate rule to apply, without any change, to send-based DDP?

I believe they do.  rtrext is written using that concept which
you created and required that ULBs define.  That gives us a good
basis to make ULBs more generic.  I just want to take
advantage of it sooner than you do.

> I don't think that's a foregone conclusion.

Perhaps not but I believe it is the case.

> will have an impact on the way ULBs are written?

Right now rfc5667 says that if a message is longer than the
size of the receivers buffer it must go in a reply chunk.  That
isn't compatible with message continuation.

Instead, it should say that if a message is longer than the longest
message that a transport can handle using sends, it should be
transferred using explicit RDMA operations.

That would leave it up to the transport to define the longest message
that can transferred by sends and how longer messages are transferred
using explicit RDMA operations, which rfc5666bis already does.

> agreed on, and then dinging me (repeatedly) for ignoring them.

I haven't done that.  If it seems to you that I have, it would help
if you specified one of these new rules.

> update when we have real documented transport changes to address.

If you want to close the book on rfc56667bis, I think I can put
off what I was planning to do.

> Otherwise rfc5667bis will never be finished.

I think you are unduly pessimistic but I think we accommodate your
desire to close on rfc5667.  I'll send separate mail on the retry issue.

> Let's design for what we have,

OK.  rfc5667bis will address what we have. limited to protocols for which
there
are implementations, and working group documents that are far advanced.

Another document will address what we have in the sense that there are
existing specifications
for the feature.  In other words it will hande version-two and rtrext.  I
think it will handle a lot
more, although you might not agree.

> clear the table, and move on. There's

Exactly.  I think we can get by with exactly one.

So what I expect to do is to submit draft-dnoveck-nfsv4-nfsulb,
illustrating my
approach to making the ULBs for NFS transport-generic.  Since it is based
on rfc5667bis, I will, with your permission, list you as a co-author.