Subject: [nfsv4] Fwd: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt

From: David Noveck <[email-masked]>
Date: Thu, 18 Aug 2016 16:23:08 -0400

This is updated and it add some vowels (and consonants too) the field and
type names.  In particular "rq" --> "req".

I'm aware that some people find "XCHAR" confusing.  If someone has an idea
for a replacement, please propose it on the list.  If the working group is
OK with it, I'll produce a -03 incorporating it.

---------- Forwarded message ----------
From: <[email-masked]>
Date: Thu, Aug 18, 2016 at 4:16 PM
Subject: New Version Notification for
draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt
To: David Noveck <[email-masked]>

A new version of I-D, draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt
has been successfully submitted by David Noveck and posted to the
IETF repository.

Name:           draft-dnoveck-nfsv4-rpcrdma-xcharext
Revision:       02
Title:          RPC-over-RDMA Extension to Manage Transport Characteristics
Document date:  2016-08-18
Group:          Individual Submission
Pages:          23
URL:            https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-
rpcrdma-xcharext-02.txt
Status:         https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-
rpcrdma-xcharext/
Htmlized:       https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-
xcharext-02
Diff:           https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-
rpcrdma-xcharext-02

Abstract:
   This document specifies an extension to RPC-over-RDMA Version Two.
   The extension enables endpoints of an RPC-over-RDMA connection to
   exchange information which can be used to optimize message transfer.

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

---

From: karen deitke <[email-masked]>
Date: Fri, 19 Aug 2016 12:46:06 -0600

How about simply PROP for property?

Karen

On 8/18/2016 2:23 PM, David Noveck wrote:
> This is updated and it add some vowels (and consonants too) the field 
> and type names.  In particular "rq" --> "req".
>
> I'm aware that some people find "XCHAR" confusing.  If someone has an 
> idea for a replacement, please propose it on the list.  If the working 
> group is OK with it, I'll produce a -03 incorporating it.
>
>
> From: <[email-masked] <mailto:[email-masked]>>
> To: David Noveck <[email-masked] <mailto:[email-masked]>>
>
>
>
> URL: 
> https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt 
> <https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt>
> Status: 
> https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-rpcrdma-xcharext/ 
> <https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-rpcrdma-xcharext/>
> Htmlized: 
> https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-xcharext-02 
> <https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-xcharext-02>
> Diff: 
> https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-rpcrdma-xcharext-02 
> <https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-rpcrdma-xcharext-02>
>
> Abstract:
> until the htmlized version and diff are available at tools.ietf.org 
> <http://tools.ietf.org>.
>
> The IETF Secretariat
>
>
>
>
> _______________________________________________
> nfsv4 mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/nfsv4

---

From: David Noveck <[email-masked]>
Date: Fri, 19 Aug 2016 15:00:04 -0400

> How about simply PROP for property?

That's OK with me.

If there is no objection, I'll incorporate that in -03.  I'll produce that
in the second week of September, so I'd appreciate it if we can get all
comments that have a significant effect on the XDR out by the end
of August.

On Fri, Aug 19, 2016 at 2:46 PM, karen deitke <[email-masked]>
wrote:

> Subject: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-
> xcharext-02.txt
> URL:            https://www.ietf.org/internet-
> drafts/draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt
> Status:         https://datatracker.ietf.org/
> doc/draft-dnoveck-nfsv4-rpcrdma-xcharext/
> Htmlized:       https://tools.ietf.org/html/d
> raft-dnoveck-nfsv4-rpcrdma-xcharext-02
> Diff:           https://www.ietf.org/rfcdiff?
> url2=draft-dnoveck-nfsv4-rpcrdma-xcharext-02
>
> Abstract:
> nfsv4 mailing [email-masked]://www.ietf.org/mailman/listinfo/nfsv4
>
>
>

---

From: Chuck Lever <[email-masked]>
Date: Fri, 19 Aug 2016 15:03:34 -0400

> On Aug 19, 2016, at 3:00 PM, David Noveck <[email-masked]> wrote:
> 
> If there is no objection, I'll incorporate that in -03.  I'll produce that in the second week of September, so I'd appreciate it if we can get all comments that have a significant effect on the XDR out by the end of August.

The only issue I feel is not resolved is the way UPDATE_XCHAR works.
I'll have another look at that in -02.

> On Fri, Aug 19, 2016 at 2:46 PM, karen deitke <[email-masked]> wrote:
>> This is updated and it add some vowels (and consonants too) the field and type names.  In particular "rq" --> "req".
>> 
>> I'm aware that some people find "XCHAR" confusing.  If someone has an idea for a replacement, please propose it on the list.  If the working group is OK with it, I'll produce a -03 incorporating it. 
>> 
>> 
>> Subject: New Version Notification for draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt
>> URL:            https://www.ietf.org/internet-drafts/draft-dnoveck-nfsv4-rpcrdma-xcharext-02.txt
>> Status:         https://datatracker.ietf.org/doc/draft-dnoveck-nfsv4-rpcrdma-xcharext/
>> Htmlized:       https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-xcharext-02
>> Diff:           https://www.ietf.org/rfcdiff?url2=draft-dnoveck-nfsv4-rpcrdma-xcharext-02
>> 
>> Abstract:

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Mon, 22 Aug 2016 14:10:59 -0400

Remarks on rpcrdma-xcharext-02.

- Credit accounting

There's no discussion of RPC-over-RDMA credit accounting in this
document. There needs to be some discussion of credit consumption.
In particular:

Requesters will have to have posted extra receive buffers (over
and above credits for forward channel replies and backchannel
requests) to deal with XCHAR messages. Likewise, responders will
need to post similar extra receives for this purpose.

Perhaps both peers should reserve one credit, and the specification
should insist that these operations are always single-threaded on a
connection.

Alternately, when xcharext is merged into rpcrdma-version-two,
there might be a generic discussion of non-RPC-payload-bearing
messages that could cover this issue.

- Overloading the term "initial"

Section 3 introduces an "initial set of transport characteristics"
and Section 4.1 defines "ROPT_INITXCH: Specify Initial
Characteristics". I think the use of "initial" means different
things in these two cases?

Instead:

Section 3 could propose "initially-defined" characteristics.

Section 4.1 could define ROPT_STARTXCH (like STARTTLS). I'm not
attached to that name, but it probably shouldn't be ROPT_INITXCH
if Section 3 defines "initial characteristics".

ROPT_CONNXCH

ROPT_FIRSTXCH

ROPT_EARLYXCH

- Section 4.4

The text in this section has been clarified to address previous
reviewer comments; thanks! There are a number of syntax and
grammatical errors that still need to be addressed (most often,
a few words are repeated in some sentences).

The "argument structure" of ROPT_UPDXCH is:

struct optinfo_updxch {
    xcharval        optupdxch_now;
    bool            optupdxch_pendclr;
};

I prefer optupdxch_new instead of optupdxch_now; "_now" suggests this
field records a time and/or date stamp.

It would help me to understand why a receiver needs to distinguish
between these types of notification.

For instance, if pendclr is false, this could be either a rejection
of a pending change request, or it could be an unsolicited change
notification. How does the receiver make use of the difference?

Instead of a boolean, an enumeration of update event types would be 
a little friendlier, and could be expressed in the same amount of
space (since an XDR boolean consumes 4 octets on the wire). Based
on the discussion in Section 4.4, we have:

enum optupdxch_event_type {
   OPTUPD_UNSOL  = 1,
   OPTUPD_MORE   = 2,
   OPTUPD_DONE   = 3,
   OPTUPD_REJECT = 4,
};

But since the rdma_xid field is not used to tie change requests to
these change update notifications, I'm not sure why the receiver
needs to know that a pending request has been completed. I think
REJECT might be more interesting than the difference between UNSOL,
DONE, and MORE.

There is a bit of a race here: a sender could send an unsolicited
update notification at the same time the receiver requests a change
of the same xchar. Could that result in a non-deterministic outcome?

Would it ever be reasonable to send two or more updates
simultaneously for the same XCHAR? (Requiring single-threading here
would prevent that from occurring).

What if the sender emits two optinfo_updxch messages: both with
pendclr set to false, but one with an intermediate value, and one
with the original value. The result on the receiver could depend on
the order in which these messages arrive. Possibly some text
regarding the ordering of these messages is needed.

What happens if the receiver of ROPT_REQXCH drops the request? Is
there a timeout after which ROPT_REQXCH may be sent again?

What happens if an ROPT_RESPXCH is dropped? If ROPT_REQXCH is sent
again, the reply is :

  ROPT_RESPXCH with the requested value marked done ?
  ROPT_RESPXCH with a rejection (no change was done) ?
  ROPT_UPDXCH with the requested value and pendclr set to true ?

I don't see language that disallows any of these responses. Which
one means "I already set this value" ? Sorry if I missed that.

Assuming that both sides support ROPT_UPDXCH, can an implementation
use ROPT_UPDXCH exclusively instead of ROPT_INITXCH?

Assuming that both sides support ROPT_UPDXCH, may a peer change an
XCHAR and not send an unsolicited ROPT_UPDXCH?

> On Aug 18, 2016, at 4:23 PM, David Noveck <[email-masked]> wrote:
> 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Tue, 23 Aug 2016 10:45:47 -0400

Thanks. I'll address these as part of
draft-dnoveck-nfsv4-rpcrdma-xcharext-03, which will be out in the first
half of September.

Unless there are objections, that draft will adopt Karen's suggestion to
change "characteristics" to "properties".  The title of
the document will change but the file name will stay the the same.

> document. There needs to be some discussion of credit consumption.

Right.

> requests) to deal with XCHAR messages.

I think you mean that clients will.

> Likewise, responders will

Similarly, I think servers are being referred to.

> should insist that these operations are always single-threaded on a

I think single-threading might be an implementation choice but I'm
reluctant to make it part of the protocol.

> messages that could cover this issue.

I think there will need to be generic text regarding the issues, but I
think it should
be consistent with the following approach regarding messages related to
transport.

In the case of messages that do not serve effectively as the response

to a previous message (i.e. ROPT_FIRSTPROP, ROPT_REQPROP,

ROPT_UPDPROP), it is the responsibility of the sender to ensure that

there is a credit available to enable sending the message, just as would

be the case if it were sending an RPC request.

In the case of ROPT_RESPPROP meesages, it is the responsibility of

the sender of the original ROPT_REQPROP to post a receive to

receive the response, which is to be sent by the receiver of the
ROPT_REQPROP,

without consuming a credit.   This is similar to the case of ending an RPC

response.

> Overloading the term "initial"

Your'e right.  It is overloaded.

> things in these two cases?

It does.

> Section 3 could propose "initially-defined" characteristics.

OK.

> ROPT_EARLYXCH

My current choice is ROPT_FIRSTPROP.

> Section 4.4

> a few words are repeated in some sentences).

I'll address those in -03.

> field records a time and/or date stamp.

OK.

> between these types of notification.

When I specified the possible situations in which these messages could
arise, I did not mean to imply that the receiver necessarily would need to
distinguish these notifications.  My focus was on the sender.

>of a pending change request,

If it it is a *rejection* of a pending change request pendclr would true

> or it could be an unsolicited change
> notification.

> How does the receiver make use of the difference?

> space (since an XDR boolean consumes 4 octets on the wire).

I'm OK in principle, but it seems we are both uncertain if this is
indeed "friendlier"

> Based

> enum optupdxch_event_type {

pendclr = false as there is nothing pending to clear.

>   OPTUPD_MORE   = 2

pemdclr=false but there is still a pending request.
,
>   OPTUPD_DONE   = 3,

pendclr = true and the request has completed successfully.

   OPTUPD_REJECT = 4,

This sound to me pendclr= true but the word "REJECT" suggests no change at
all was made.  In this case you might also have a partial change
};

> needs to know that a pending request has been completed.

If the receiver keeps track of pending requests, it needs to know when one
is no longer pending.

The receiver is not required to do this and some might not choose to do so,
but the protocol should provide
an implmentation that does the means to keep track.

This does not require the xid, but peer that requested the change can keep
track of the properites for
which it has a pending requested change.

> I think

How about an enum that distinguished:

   - Unsolicited
   - Clear pending
   - Still Pending

The additional distinction among degrees of request satisfaction in the
last two
cases could be the responsibility of the receiver to determine. Since he
made the request
and has access to the current value, he could determine this himself.
Alternatively,
we could have an int with two distinct bit fields:

   - One distinguishing unsolicited/clear-pending/still-pending
   - Another regarding degrees of request satisfaction.

> of the same xchar. Could that result in a non-deterministic outcome?

It shouldn't.  The point is that the property changes happen in a sequence
which
is the same for all observers (no weird relativistic effects!) and that the
updates
to the peer should happen that same sequence.  The important point is a
RESPXCH indicating a succesful change request be in the proper place in
that
sequence.

> simultaneously for the same XCHAR?

Since this is a single connection with sequenced delivery, there is
no way to send updates simultaneously.  You can queue them for
sending at the same time, but the delivery will reflect the order in
which were queued.

> (Requiring single-threading here would prevent that from occurring).

Given that there is no response to UPDXCH, not sure how you
could specify single-threading.  There is no way to define when
it would be OK to send the next.

> the order in which these messages arrive.

It would.

> Possibly some text

I can add something.

> What happens if the receiver of ROPT_REQXCH drops the request?

>Is there a timeout after which ROPT_REQXCH may be sent again?

There is no timeout in the protocol.

An implementation may choose to do so, but since this is sent
on a reliable connection, it is hard to imagine it being worth doing.

>  ROPT_RESPXCH with the requested value marked done ?

This should result.

>   ROPT_RESPXCH with a rejection (no change was done) ?

It is true no change was done but requested value was achieved,

>  ROPT_UPDXCH with the requested value and pendclr set to true ?

ROPT_UPPDXCH is not an alternative to ROPT_RESPXCH.  It is
a possible additional message.

> one means "I already set this value" ? Sorry if I missed that.

I can add some clarification.

> use ROPT_UPDXCH exclusively instead of ROPT_INITXCH?

Yes but it  is kind of bogus.  You would be relying on the default initial
values
and then changing them, which would be good in a test but, in real life, it
is
asking for trouble.

BTW, I've always wished there would be an RFC2119bis defining "BOGUS" and
"BRAIN-DEAD" :-)

> XCHAR and not send an unsolicited ROPT_UPDXCH?
It may but in most cases it would be BOGUS (or BRAIN-DEAD).

Suppose you raise the receive buffer size and don't tell you peer that it
is raised,  In
that case. raising it is pretty pointless since the peer can't take
advantage of the bigger
buffer.

If you lower the receive buffer size and don't tell the peer it has been
lowered, then he
is going to continue to assume a larger size and break things.

---

From: karen deitke <[email-masked]>
Date: Tue, 23 Aug 2016 09:56:27 -0600

One additional thing that has come up is the lack of a keepalive type 
activity on ib.  We've had some discussions with the infiniband driver 
folks as to where this could be done, hoping that it could be done at 
the lower transport level as is with tcp,and not in rpc/rdma area.  The 
issue is that if the client sends a request tot he server and is waiting 
for a response and the server crashes, and there is no other activity 
going on, we will wait forever, or until some other activity is 
initiated on the connect before we know that the connection is gone.

The suggestion was made that the client periodically issue a 0 length 
write.  This would not take up a recv buffer if the other side is still 
up, and can detect that the connection is gone in this situation.  
However the  ib layer folks do not want to implement this in their side, 
as they don't have the qp and cqp info of our configuration.  Their 
concern is, for example, if the 0 length write fails, it will still use 
a response in the completion queue, which could potentially, if full, 
result in a valid connection being lost if active.  Seems this wouldn't 
be the case for an inactive connection, but its possible that the rcq is 
a system wide shared rcq also.  Because of issues like this, the ib 
driver folks are recommending we do this in the rpc layer.

So, wondering, is/should this be something addressed in this version of 
rpcrdma?

Karen

On 8/23/2016 8:45 AM, David Noveck wrote:
> draft-dnoveck-nfsv4-rpcrdma-xcharext-03, which will be out in the 
> first half of September.
>
> Unless there are objections, that draft will adopt Karen's suggestion 
> to change "characteristics" to "properties".  The title of
> >    xcharvaloptupdxch_now;
> >    booloptupdxch_pendclr;
> arise, I did not mean to imply that the receiver necessarily would 
> need to distinguish these notifications.  My focus was on the sender.
>
> This sound to me pendclr= true but the word "REJECT" suggests no 
> change at all was made.  In this case you might also have a partial change
> };
>
> If the receiver keeps track of pending requests, it needs to know when 
> one is no longer pending.
>
> The receiver is not required to do this and some might not choose to 
> do so, but the protocol should provide
> This does not require the xid, but peer that requested the change can 
> keep track of the properites for
>   * Unsolicited
>   * Clear pending
>   * Still Pending
> The additional distinction among degrees of request satisfaction in 
> the last two
> cases could be the responsibility of the receiver to determine. Since 
> he made the request
>   * One distinguishing unsolicited/clear-pending/still-pending
>   * Another regarding degrees of request satisfaction.
>
>
>
> It shouldn't.  The point is that the property changes happen in a 
> sequence which
> is the same for all observers (no weird relativistic effects!) and 
> that the updates
> RESPXCH indicating a succesful change request be in the proper place 
> in that
> sequence.
>
> >Isthere a timeout after which ROPT_REQXCH may be sent again?
>
> Yes but it  is kind of bogus.  You would be relying on the default 
> initial values
> and then changing them, which would be good in a test but, in real 
> life, it is
> asking for trouble.
>
> Suppose you raise the receive buffer size and don't tell you peer that 
> it is raised,  In
> If you lower the receive buffer size and don't tell the peer it has 
> been lowered, then he
> On Mon, Aug 22, 2016 at 2:10 PM, Chuck Lever <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     > On Aug 18, 2016, at 4:23 PM, David Noveck <[email-masked]
>     <mailto:[email-masked]>> wrote:
>     >
>     > This is updated and it add some vowels (and consonants too) the
>     field and type names.  In particular "rq" --> "req".
>     >
>     > I'm aware that some people find "XCHAR" confusing. If someone
>     has an idea for a replacement, please propose it on the list.  If
>     the working group is OK with it, I'll produce a -03 incorporating it.
>     >
>     >
>     https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-xcharext-02 <https://tools.ietf.org/html/draft-dnoveck-nfsv4-rpcrdma-xcharext-02>
>     > Diff:
>     > until the htmlized version and diff are available at
>     tools.ietf.org <http://tools.ietf.org>.
>     >
>     > The IETF Secretariat
>     >
>     >
>     > [email-masked] <mailto:[email-masked]>
>     <https://www.ietf.org/mailman/listinfo/nfsv4>
>
>     --
>     Chuck Lever
>
>
>
>
>
>

---

From: Chuck Lever <[email-masked]>
Date: Tue, 23 Aug 2016 12:57:38 -0400

> On Aug 23, 2016, at 10:45 AM, David Noveck <[email-masked]> wrote:
> 
> Thanks. I'll address these as part of draft-dnoveck-nfsv4-rpcrdma-xcharext-03, which will be out in the first half of September.
> 
> Unless there are objections, that draft will adopt Karen's suggestion to change "characteristics" to "properties".  The title of
> I think there will need to be generic text regarding the issues, but I think it should
> receive the response, which is to be sent by the receiver of the ROPT_REQPROP,
> When I specified the possible situations in which these messages could arise, I did not mean to imply that the receiver necessarily would need to distinguish these notifications.  My focus was on the sender.

> This sound to me pendclr= true but the word "REJECT" suggests no change at all was made.  In this case you might also have a partial change
> };
> 
> If the receiver keeps track of pending requests, it needs to know when one is no longer pending.
> 
> The receiver is not required to do this and some might not choose to do so, but the protocol should provide
> This does not require the xid, but peer that requested the change can keep track of the properites for

So, there is a protocol mechanism for reporting partial or full
completion of a property change request, yet no mechanism for
indicating which change request it is the receiver is completing.
But multiple property change requests are allowed to be in flight
concurrently.

Without an XID, the only way this might work is by serializing
updates to each property.

And, if the sender can send an "I'm done" update followed by any
number of "Oops, I changed this again" updates, I don't see the
point of a receiver ever remembering that it is waiting for a
pending change, or the protocol distinguishing between unsolicited
and pendclr.

That makes pendclr largely a hint, and a temporary one at that.

> 	• Unsolicited
> 	• Clear pending
> 	• Still Pending

> The additional distinction among degrees of request satisfaction in the last two
> cases could be the responsibility of the receiver to determine. Since he made the request
> and has access to the current value, he could determine this himself.  Alternatively,
> 	• One distinguishing unsolicited/clear-pending/still-pending
> 	• Another regarding degrees of request satisfaction.

In other words "rejection" doesn't matter. There's no recourse for
the receiver; it just needs to know the current value of the
property.

> It shouldn't.  The point is that the property changes happen in a sequence which
> is the same for all observers (no weird relativistic effects!) and that the updates
> RESPXCH indicating a succesful change request be in the proper place in that 
> sequence.
> 

If two or more ULPs are using the same connection, they might
send a sequence of possibly contradictory property change
requests.

Also, there's no co-ordination between the two senders on the
same connection; or an administrator might adjust these
properties.

I posit that the receiver cares only about the current effective
value of these properties. pendclr is completely unreliable.

Can you provide a real world example of how pendclr might be used?
Because it appears to me to convey no actionable information, and
I don't see why I shouldn't always set it to false.

Think of this as an NFS attribute: SETATTR doesn't return "Oh, I
changed this just a little" or "Someone adjusted this value, but
it may change again".

> on a reliable connection, it is hard to imagine it being worth doing.

Does the document's language forbid a receiver from simply discarding
a property change request? (I'm thinking of similar language as RFC
7350 which requires that NFS servers never discard requests).

> Yes but it  is kind of bogus.  You would be relying on the default initial values
> and then changing them, which would be good in a test but, in real life, it is
> asking for trouble.

It's very easy to imagine an implementer deciding that UPDXCH is
better for his or her client or server than FIRSTPROP, and noting
stridently that the document does not prohibit or even warn against
this design.

IMO the document needs to make an explicit requirement not to
behave this way, or state that it is acceptable though not preferred.

> BTW, I've always wished there would be an RFC2119bis defining "BOGUS" and

The closest we have is SHOULD NOT (which I always read as "don't do
that, you muppet").

> Suppose you raise the receive buffer size and don't tell you peer that it is raised,  In
> that case. raising it is pretty pointless since the peer can't take advantage of the bigger 
> buffer.
> 
> If you lower the receive buffer size and don't tell the peer it has been lowered, then he

For a client to reduce its receive buffers in mid-flight, all it
needs to do is send Reply chunks more often. That will work OK
if the server uses the Reply chunk whenever one is provided.
(Not all servers behave nicely in this regard, though).

For a server to reduce its receive buffers in mid-flight, the
server needs to notify the client of the change in receive
buffer size, the client would need to acknowledge the change,
and only then can the server reduce its receive buffer size.
How might that be done with the protocol proposed in
rpcrdma-xcharext ?

OK imagine some less harmful changes, like:

The administrator on the client has turned a knob that enables
Remote Invalidation. Yet it is perfectly safe for the server to
continue not using RI. Is UPDXCH required in that case?

If you'd rather the document does not commit, then maybe a
sentence or two could be added that suggests the implementer
needs to consider carefully the safety of not sending an
unsolicited property change.

--
Chuck Lever

---

From: Chuck Lever <[email-masked]>
Date: Tue, 23 Aug 2016 13:10:47 -0400

Hi Karen-

> On Aug 23, 2016, at 11:56 AM, karen deitke <[email-masked]> wrote:
> 
> One additional thing that has come up is the lack of a keepalive type activity on ib.  We've had some discussions with the infiniband driver folks as to where this could be done, hoping that it could be done at the lower transport level as is with tcp,and not in rpc/rdma area.  The issue is that if the client sends a request tot he server and is waiting for a response and the server crashes, and there is no other activity going on, we will wait forever, or until some other activity is initiated on the connect before we know that the connection is gone.
> 
> The suggestion was made that the client periodically issue a 0 length write.  This would not take up a recv buffer if the other side is still up, and can detect that the connection is gone in this situation.  However the  ib layer folks do not want to implement this in their side, as they don't have the qp and cqp info of our configuration.  Their concern is, for example, if the 0 length write fails, it will still use a response in the completion queue, which could potentially, if full, result in a valid connection being lost if active.  Seems this wouldn't be the case for an inactive connection, but its possible that the rcq is a system wide shared rcq also.  Because of issues like this, the ib driver folks are recommending we do this in the rpc layer.
> 
> So, wondering, is/should this be something addressed in this version of rpcrdma?

Have a look in the [email-masked] mail archive for a thread entitled

  "Detecting loss of connection with RPC-over-RDMA"

that starts on January 26, 2016.

The zero-length Write approach was largely rejected by the WG in favor
of retaining a single spare RPC-over-RDMA credit for high-priority
requests.

However, I think the real problem is that the hardware retry count is
set to its maximum, which for some HCAs means it continues retrying
message transmission for a rather indefinitely long period of time.
Lowering that retry maximum might achieve what you need; that approach
was quite successful on the Linux client.

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 26 Aug 2016 05:08:48 -0400

> But multiple property change requests are allowed to be in flight

The protocol allows it, but particular implementations, may well
restrict things so that they have only one in flight for each property
at any particular time.

> updates to each property.

It is one way.  I don't think it is the only way.  The requester, if it
is interested in knowing when there are no outstanding requests
for a property can increment on requests and decrement on messages
that indicate completion.

> number of "Oops, I changed this again" updates,

The updates indicate that there has been an unsolicited change in
the property:  no "Oops" and no "again".

Whether such changes happen or not, depends on the
implementation.  In any case, I would expect them to be
rare.

The proposed protocol provides a way for the peer to be
informed of such changes.   I don't see how that undercuts
the facilities that allow the completion of requested changes
to be reported asynchronously.

> I don't see the

You seem to be saying the potential existence of unsolicited changes
some time after the completion of a requested change somehow
makes the notion of completion of the requested change useless.

I'm not sure why you believe that.  The protocol allows unsolicited changes
but that doesn't mean they are going to be happening all that frequently.

> That makes pendclr largely a hint, and a temporary one at that.

I don't see things that way.

>In other words "rejection" doesn't matter.

Yes, but the indication that the request is done can matter, even if it
might not
matter to you.  See below.

> There's no recourse for the receiver;

I disagree.

> it just needs to know the current value of the property.

It certainly needs to know that.  Your implementaition might
not be interested in anything else but the protocol provides it
because it is easy for the sender to provide and useful in some
cases.

> send a sequence of possibly contradictory property change

I don't understand why one might assume that ULP's
are sending property change requests.  ULPs are RPC
protocols.  They send rpc requests and receive replies.

> same connection; or an administrator might adjust these

Any co-ordination requirement that a transport implementation
might impose to deal with the case of multiple ULPs would be
out of scope in this document.

I don't see how you decide there is "no co-ordination".  If you build
an implementation and give multiple ULPs the ability to request and
make such changes independently, you are asking for (or begging for)
trouble.  As an implementer, the choice regarding co-ordination (or not)
is up to you.

I think the entities making such changes are most likely to reflect
administrative requests or internal transport optimization functions.

I can't see why one would allow the ULP to do this itself.

> value of these properties.

That's one of the things he cares about, but I believe it is not
the only one.

pendclr is completely unreliable.

If you were to always set it to false, as you suggest below that you
might, you can make it unreliable, but there is no good reason to do that.

> I don't see why I shouldn't always set it to false.

Let's consider that a client implementation encounters a server
which has a very large receive buffer size and feels that it would prefer
more
smaller buffers.

It then requests a smaller receiver buffer size.  as part of doing that it
decides
to not send requests or responses that are larger than this anticipated size
limit.  I think the spec already discusses this but the basic point is that
by making
this request the sender is licensing the peer to reduce the buffer size.
If this
request is completed, the sender will make that new size permanent (or at
least until
it hears about a change). On the other hand, if the peer rejects the
request
(either fairly soon via RESPXCH, or  after a while via UPDXCH), the sender
would
then adopt the current value, since the possibility of the
peer eventually
satisfying the orginal request is now gone. On the other hand, an update
not including pendclr
should not have that effect.  As long as the request is pending, he has to
be prepared
for its peer to adopt the lower value that it has requested.

> Think of this as an NFS attribute

The point is that it isn't an NFS attribute, even though there are some
similarities.

Let's look at the differences and the reasons for them.

   - SETATTR does not have UPDXCH since there is no way to have a
   potentially long-running change requiring asychrnous notification.  SETATTR
   generally cannot take significant time.
   - Although there are attributes subject to unrequested change, you are
   not informed about them.  for example space-available

> SETATTR doesn't return "Oh, Ichanged this just a little"

> "Someone adjusted this value, but it may change again"

It doesn't tell you that but you have no guarantee that someone else will
not change
after you.  In the model we are talking about, it also may change but you
receive a
notification.  That sort of notification would make not sense for an NFSv4
attribute
but it does make sense about a transport property because it is a
reasonable assumption that
the sender wants to know about this ASAP since it might affect his next
transfer

> 7350 which requires that NFS servers never discard requests).

I can add something like that.

> behave this way, or state that it is acceptable though not preferred.

Are you sure that it is "requirement' and not a "REQUIREMENT" that you
are looking for?

I'm OK with saying you shouldn't do that, but I have problem saying it is
"acceptable".  I think it is noxious, although there is no "NOXIOUS".

Maybe the right thing is to state the obvious, that the function of UPDXCH
is to
convey changes, making its use to present values at connection time dubious,
while the function of ROPT_FIRSTPROP is to present the value at initial
connection.

> The closest we have is SHOULD NOT (which I always read as "don't do

I know that is a common usage/interpretation but it is not consistent with
RFC2119.
Lately the IESG has started to notice that.

> How might that be done with the protocol proposed in

I don't think unsolicited reduction in receive buffer size (as opposed to
the
requested reduction discussed above) can be made to work.
Maybe I should add an explicit statement
to that effect.

> continue not using RI. Is UPDXCH required in that case?

No.

OK.

On Tue, Aug 23, 2016 at 12:57 PM, Chuck Lever <[email-masked]>
wrote:

> > Thanks. I'll address these as part of draft-dnoveck-nfsv4-rpcrdma-xcharext-03,
> which will be out in the first half of September.
> >
> > This sound to me pendclr= true but the word "REJECT" suggests no change
> at all was made.  In this case you might also have a partial change
> > };
> >

---

From: Chuck Lever <[email-masked]>
Date: Fri, 26 Aug 2016 11:39:21 -0400

Hiya Dave-

> On Aug 26, 2016, at 5:08 AM, David Noveck <[email-masked]> wrote:
> 

No, I'm saying that pendclr adds no value; the receiver knows
everything it needs to know by looking only at the new value.

I don't have a problem with UPDXCH as a mechanism for unsolicited
change notifications, though I'm not sure we have a use case
where the same kind of notification cannot be done with REQXCH.

> I'm not sure why you believe that.

I believe that, because I can't for the life of me think of a use
case where a receiver can make use of the information in pendclr.
(Yes, I probably lack quite a bit of imagination). Can you please
provide one?

It's difficult, as a reviewer of proposed protocol, to understand
why we are reserving these 32 bits in the UPDXCH message without
having at least a little clue about why they are there and how a
receiver would make use of them.

It's especially perplexing to me to see a facility for asynchronous
notification of completion when there is no real call-reply transaction
relationship between REQXCH and REPXCH, and there is no timeout when
sending REQXCH messages. A key question for me is "How is [UPDXCH
with pendclr set] different than a delayed REPXCH response?" If they
are semantically the same, then why have both?

If the document moves forward with pendclr, I think implementers need
more guidance about how to use this field. Otherwise, someone is going
to build an implementation that depends on pendclr, and some other
implementation that implemented the protocol blindly (ie without any
need for the pendclr value) is not going to get it right (or will take
shortcuts, or whatever), and there will be an interoperability problem,
just add boiling water and fluff with fork.

In other words, I think there is useful feedback from an implementation
that does not need pendclr. Perhaps only interop testing can sort it
out.

> The protocol allows unsolicited changes

A sane implementation of the currently proposed initial properties
probably won't use UPDXCH at all. I'm trying to explore the corners
here and play devil's advocate; if an implementer is allowed to use
UPDXCH in a crazy way, we should try to anticipate that and either
prohibit it or make it safe.

> because it is easy for the sender to provide and useful in some

To interoperate among implementations that depend on it and
ones that don't need it, though, I think the document needs
more precise language about how pendclr is to be used.

> protocols.  They send rpc requests and receive replies.

(Much) earlier you suggested that a ULP might want to select its
own receive buffer size, for example. If two ULPs share the same
connection, then they could choose to adjust the connection in
contradictory ways. Perhaps you are taking that off the table
now, which is fair enough.

> out of scope in this document.

My remark here was not about two ULPs. The two directions of
transmission on a connection are not co-ordinated in any way. An
unsolicited UPDXCH can pass a REQXCH going in opposite directions.

IMO the document has to acknowledge that property value changes
have to be done carefully and with co-ordination with the regular
operation of the transport to avoid hazardous oscillations of
values. The PROP protocol mechanism does not provide any kind of
serialization on its own.

> I can't see why one would allow the ULP to do this itself.

Yes, I'm talking about administrative requests or optimization
efforts, here.

> which has a very large receive buffer size and feels that it would prefer more 
> smaller buffers.

More likely, either the client or server is low on resources, and
wishes to reclaim receive buffer space. But, OK.

> It then requests a smaller receiver buffer size.  as part of doing that it decides
> limit.  I think the spec already discusses this but the basic point is that by making
> this request the sender is licensing the peer to reduce the buffer size.  If this
> request is completed, the sender will make that new size permanent (or at least until
> it hears about a change). On the other hand, if the peer rejects the request 
> (either fairly soon via RESPXCH, or  after a while via UPDXCH), the sender would
> satisfying the orginal request is now gone. On the other hand, an update not including pendclr
> should not have that effect.  As long as the request is pending, he has to be prepared

I don't see how the REQuester's behavior would be different if pendclr
was false. It can't assume the requested setting takes effect until
it gets a REPXCH or UPDXCH with the new value, whether or not pendclr
is true, and there's nothing in the spec (that I recall) that says
otherwise (about pendclr being true is required before a new value is
trusted -- did I miss something? -- and if that is the case, then
what's the value of sending an unsolicited UPDXCH ?).

There's nothing that prevents the REP side from sending "yes, I took
the new setting (pendclr true)" and then "oops, no I went back to the
old one (pendclr false)". That's not something an implementer might
do on purpose, but it sure can happen by accident (poorly-timed
administrator action, say).

> I'm OK with saying you shouldn't do that, but I have problem saying it is "acceptable".  I think it is noxious, although there is no "NOXIOUS".
> 
> Maybe the right thing is to state the obvious, that the function of UPDXCH is to
> while the function of ROPT_FIRSTPROP is to present the value at initial connection.

You could state that UPDXCH and REQXCH MUST/SHOULD NOT be used before
FIRSTPROP has been exchanged.

Or even should not. I would go with MUST NOT: then receivers can depend
on seeing FIRSTPROP before seeing other changes.

> I don't think unsolicited reduction in receive buffer size (as opposed to the

It could, perhaps, if the property had two values: one was the send
size limit, and the other was the receive buffer size. The send size
limit is not necessarily associated with a physical buffer size, if
Sends are done with a gather vector, so its relatively
straightforward to change it.

One peer could send a REQXCH to reduce the sender's send size limit.
If the other peer says "OK" then it is safe to reduce the receive
buffer size.

In fact, IMO both sides do want to know the send size limit of their
peer so they can trim receive buffers to a size that would be used.
It might also help a requester determine when a responder needs a
Reply chunk.

> Maybe I should add an explicit statement

That would be implementation guidance at best, and might preclude
some future innovation that allows unsolicited receive buffer size
changes.

Should similar statements be made about the other initial properties
proposed in rpcrdma-xcharext? 

--
Chuck Lever

---

From: David Noveck <[email-masked]>
Date: Fri, 26 Aug 2016 13:57:16 -0400

> everything it needs to know by looking only at the new value.

That seems to be because, you believe that the status of a
previously pending request (completed or not) is not something
that the receiver needs to know.  In some cases it might
not but given that there is mechanism to make requests, it
seems to me that the one making the request

> where the same kind of notification cannot be done with REQXCH.

REQXCH cannot be used for unsolicited change notifications.  It
is intended for  requested/solicited changes.

> are semantically the same, then why have both?

It is possible to limit UPDXCH to unsolicited changes.  The issue I
have to consider is how to deal with partial satisfaction (e.g.  you ask
for 32K and get 16K).

> more guidance about how to use this field.

Let me think about the text for this.

> Otherwise, someone is going
> to build an implementation that depends on pendclr,

That presumably would be an implementation that requested a property change.

> and some other
> need for the pendclr value)

The implementation it would be operating with would be one that
responded to such requests.  And an implementation that responds to
such requests and doesn't make them has no need for the pendclr,
but it still should set it for those that need it.

> that does not need pendclr.

I'm not sure what that feedback would be.  The fact implementation A has
no use for it does not mean that information B has no need for it.

> Perhaps only interop testing can sort it out.

Perhaps.

> probably won't use UPDXCH at all.

I don't see much use of UPDXCH (by sane implementations) for a while but
i think there will eventually be a need for it.

> I'm trying to explore the corners

I don't think safety is the issue,  Even if it is safe, people using it
the way you anticipate would disruptive, in that other implementations
would be forced to implement the same function twice, to work
with an implementation

> more precise language about how pendclr is to be used.

I'll try to provide that.

> now, which is fair enough.

Consider it off the table.

> unsolicited UPDXCH can pass a REQXCH going in opposite directions.

Yes.

> values.

Agree.

> The PROP protocol mechanism does not provide any kind of

True.

request

> trusted -- did I miss something? --

It's not that it is not trusted.  It is that when you have a request
outstanding to redeuce the the
size to X, you need to take that account.  So, if you get an update that
the size is now Y, you
accept that the size is Y when that was sent but the possibility exists
that it might be set
down to X, since you requested that that be done.

> > and if that is the case, then
> > what's the value of sending an unsolicited UPDXCH ?)

In general there is value in doing so, but there may be special
circumstances in which a
existence of a pending  request restricts the receiver from acting on it. .

> administrator action, say).

Yes, but in that case, the requester will find out that his request has
been granted,
and then that administrative action has superseded it.  the implementation
should
work OK although people might have a problem with the administrator's
choices.

> FIRSTPROP has been exchanged.

MUST NOT is OK with me.  The next step is to state the same regarding any
particular property.  I'm OK with SHOULD NOT for that.

> It could, perhaps, if ...

I was speaking of the protocol currently described.

> the property had two values: one was the send
> size limit, and the other was the receive buffer size.

I think that work better as two separate properties.  Defining a
new property
is a lot easier than changing the structure of an existing one.

> The send size

> some future innovation that allows unsolicited receive buffer size

I think I could draft something that would not run into the preclusion
problem.

You just have to point out that if the sendr is told that the receivers
buffer are X
long, he is going to be sending messages that long, making a reduction in
receive buffer size problmatic in the absence of a mechasim to nsure that
the
sender aware of the new (lower) limit.

> proposed in rpcrdma-xcharext?

Probably.

On Fri, Aug 26, 2016 at 11:39 AM, Chuck Lever <[email-masked]>
wrote:

> > I'm OK with saying you shouldn't do that, but I have problem saying it
> is "acceptable".  I think it is noxious, although there is no "NOXIOUS".
> >

---

From: karen deitke <[email-masked]>
Date: Fri, 26 Aug 2016 17:57:49 -0600

Does the receiver have any input on the unsolicited xchar update? Lets 
say the sender (i.e. NFS server for example), wants to increase its recv 
buffer size, but the receiver (NFS Client) is low on memory.  Is there 
any way for it to say I don't like that value, not now, rejected, etc?

Also, not sure if this was discussed, but if a sender or receiver 
support changes to properties, seems like they must at least have 1 
additional recv buffer posted to accept the reqxch, is one enough?

If a reqxch were sent to make a change, and the receiver believes the 
property can be changed, just maybe not to the value the sender wants, 
it needs to return the respxch with rejected, and then follow up with an 
updxch to increase the value possible just a little bit more, just not 
to the value the sender wanted.  Seems like maybe to avoid two messages, 
the receiver could have returned a respxch but indicated that new 
value.  But I don't see a value in the return in respxch.

If reqxch can only request 1 xcharspec, then why does respxch have the 
ability to indicate more than one property were changed, or 
rejected,etc?  Is this to allow multiple reqxch requests to be notified 
of completion with 1 respxch?
If the respxch were one for one with the reqxch, seems we could get rid 
of the concept of pending reqxch, especially if the respxch had the 
value which could eliminate the updxch?  Or can you put more than one 
reqxch in one messages?  That might be nice.

Also, why would an updxch be used at all, wouldn't the peer just send 
its own reqxch?  Thats whats confusing to me.  Is it more like I'm not 
asking to change this propery, I'm telling you its changed?

Also, what guarentees that the xid of the updxch is unique among the 
xid's the peer is using?

What if a reqxch and an updxch pass at the same time for the same 
property?  The updxch pendclr maybe true cause it never saw the reqxch, no?

Would it make sense with the optinfo_initxch to also have a supported 
max/min value that would possible be accepted if a change were to be made?

Also seems that the updxch's optupdxch_now (new) should be xcharspec.  
Is see that xcharspec is typedef'ed to xcharval and its really the same 
thing, but it makes for easier reading when the reqxch is an xcharspec.

Karen

On 8/26/2016 11:57 AM, David Noveck wrote:
> size to X, you need to take that account.  So, if you get an update 
> that the size is now Y, you
> accept that the size is Y when that was sent but the possibility 
> exists that it might be set
> Yes, but in that case, the requester will find out that his request 
> has been granted,
> and then that administrative action has superseded it.  the 
> implementation should
> You just have to point out that if the sendr is told that the 
> receivers buffer are X
> receive buffer size problmatic in the absence of a mechasim to nsure 
> that the
> On Fri, Aug 26, 2016 at 11:39 AM, Chuck Lever <[email-masked] 
>     > On Aug 26, 2016, at 5:08 AM, David Noveck <[email-masked]
>     > I'm OK with saying you shouldn't do that, but I have problem
>     saying it is "acceptable".  I think it is noxious, although there
>     is no "NOXIOUS".
>     >

---

From: David Noveck <[email-masked]>
Date: Sun, 28 Aug 2016 15:20:11 -0400

> Does the receiver have any input on the unsolicited xchar
> update?

No.  It is an unsolicited one-way message.  Reporting to
the receiver what has occurred.

> Lets say the sender (i.e. NFS server for example), wants to
> increase its recv buffer size, but the receiver (NFS Client) is
> low on memory.

I don't see how the fact that the client is low on memory
affects the server.  The client does not have to use the server's
larger buffer size.  He only has to make sure that he doesn't send anything
bigger than the receive buffer size.  If he is
aleady doing that for receive buffer size X, then there should be no
problem doing it for  receive buffer size Y, with Y > X.

> Is there any way for it to say I don't like that value, not now,
> rejected, etc?

No.

> Also, not sure if this was discussed, but if a sender or
> receiver support changes to properties, seems like they must
> at least have 1 additional recv buffer posted to accept the
> reqxch, is one enough?

Chuck did raise this issue.  My plan for -03 is basically to say
that a REQXCH_RESPXCH pair, is, for purposes of credit management, to be
treated as if it were

> If a reqxch were sent to make a change, and the receiver
> believes the property can be changed, just maybe not to the > value the
sender wants, it needs to return the respxch with
> rejected, and then follow up with an updxch to increase the
> value possible just a little bit more, just not to the value the
> sender wanted.

Actually the requested is not rejected.  The requested charcteristic is
included in optrespxch_pend nd there is a UPDXCH to follow up with the
selected value.

> Seems like maybe to avoid two messages, the receiver
> could have returned a respxch but indicated that new value. > But I don't
see a value in the return in respxch.

There isn't one now but I am considering a change to UPDXCH in -03 that
would include this.  The goal would not be to reduce the message count,
although it would do that.  Instead, the idea would to separate solicited
and unsolicited
changes and get rid of confusion/controversy regarding pendclr.

> If reqxch can only request 1 xcharspec, then why does
> respxch have the ability to indicate more than one property
> were changed, or rejected,etc?

one xcharspec --> multiple xcharvals so changes in multiple properties can
be requested.

> Also, why would an updxch be used at all, wouldn't the peer > just send
its own reqxch?  Thats whats confusing to me.  Is it > more like I'm not
asking to change this propery, I'm telling
> you its changed?

It is exactly the latter.

Clearly the existing text isn't clear enough.  Do you have any suggestions
to make this clearer?

> Also, what guarentees that the xid of the updxch is unique
> among the xid's the peer is using?

There aren't any and I don't believe any are needed.  There is some text
that says xid for updxch is not tied to the rqxch, but I think it need to
be more clearly stated that there is no use for this.  I could either tell
the receiver to ignore this or tell the sender to always make this zero.

> What if a reqxch and an updxch pass at the same time for
> the same property?

Do you have a specific issue in mind? What would hppen depens on the
particulars of the type of change, whether
the change

> The updxch pendclr maybe true cause it never saw the
> reqxch, no?

> Would it make sense with the optinfo_initxch to also have a > supported
max/min value that would possible be accepted if > a change were to be made?

It's possible but this is much bigger than the sort of changes I can
consider for -03.  Let's leave this for -04 or after this is merge into
version-twp

One issue with this is it assume all properties can be ordered linearly
with a min and max.  There are propertes in -rtrext that don't follow that
model

On Fri, Aug 26, 2016 at 7:57 PM, karen deitke <[email-masked]>
wrote:

> buffer size, but the receiver (NFS Client) is low on memory.  Is there any
> way for it to say I don't like that value, not now, rejected, etc?
>
> Also, not sure if this was discussed, but if a sender or receiver support
> changes to properties, seems like they must at least have 1 additional recv
> buffer posted to accept the reqxch, is one enough?
>
> property can be changed, just maybe not to the value the sender wants, it
> needs to return the respxch with rejected, and then follow up with an
> updxch to increase the value possible just a little bit more, just not to
> the value the sender wanted.  Seems like maybe to avoid two messages, the
> receiver could have returned a respxch but indicated that new value.  But I
> don't see a value in the return in respxch.
>
> ability to indicate more than one property were changed, or rejected,etc?
> Is this to allow multiple reqxch requests to be notified of completion with
> 1 respxch?
> If the respxch were one for one with the reqxch, seems we could get rid of
> the concept of pending reqxch, especially if the respxch had the value
> which could eliminate the updxch?  Or can you put more than one reqxch in
> one messages?  That might be nice.
>
> Also, why would an updxch be used at all, wouldn't the peer just send its
> own reqxch?  Thats whats confusing to me.  Is it more like I'm not asking
> to change this propery, I'm telling you its changed?
>
>
> Also, what guarentees that the xid of the updxch is unique among the xid's
> the peer is using?
>
> Also seems that the updxch's optupdxch_now (new) should be xcharspec.  Is
> see that xcharspec is typedef'ed to xcharval and its really the same thing,
> but it makes for easier reading when the reqxch is an xcharspec.
>
>
>
> Karen
>
>
>
>
>> > On Aug 26, 2016, at 5:08 AM, David Noveck < <[email-masked]>
>> [email-masked]> wrote:
>> >

---

From: David Noveck <[email-masked]>
Date: Sun, 28 Aug 2016 15:24:45 -0400

What I just sent was an incomplete reply.  You might wait for the real
reply. Not sure whether this was due to a badly designed gmail mobile
interface, my own clumsiness, or both.

On Aug 28, 2016 3:20 PM, "David Noveck" <[email-masked]> wrote:

> larger buffer size.  He only has to make sure that he doesn't send
> anything bigger than the receive buffer size.  If he is
> > could have returned a respxch but indicated that new value. > But I
>> of the concept of pending reqxch, especially if the respxch had the value

---

From: David Noveck <[email-masked]>
Date: Sun, 28 Aug 2016 19:10:53 -0400

I don't see how the fact that the client is low on memory
affects the server.  The client does not have to use the server's
larger buffer size.  He only has to make sure that he doesn't send anything
bigger than the receive buffer size.  If he is
already doing that for receive buffer size X, then there should be no
problem for him doing it for  receive buffer size Y, with Y > X.

No.

There aren't any and I don't believe any are needed.  There is some text
that says xid for updxch is not tied to that for the reqxch, but I think it
need to be more clearly stated that there is no use for this.  I could
either tell the receiver to ignore this or tell the sender to always make
this zero.

> What if a reqxch and an updxch pass at the same time for

Do you have a specific issue in mind? What would happen depends on the
particulars of the type of change and whether
the unsolicited change move things in the same or a different direction
than the change request.

It's possible but this is much bigger than the sort of changes I can
consider for -03.  Let's leave this for -04 or after this is merged into
the version-two document.

One issue with this is it assumes all possible property values can be
ordered linearly with a min and max.  For example,there here are properties
in rpcrdma-rtrext that don't follow that model.

>Also seems that the updxch's optupdxch_now (new)should be xcharspec.

I think xcharval is right

> Is see that xcharspec is typedef'ed to xcharval and its really the same
thing,

They are not supposed to be the same thing.

My intention was that xcharspec be an array of xcharvals.

Loooking at the typedef, it appears I messed that up.

I'll fIx in -03.

> but it makes for easier reading when the reqxch is an xcharspec.

I think it would help to rename xcharspec to be xcharvalset.

---

From: Chuck Lever <[email-masked]>
Date: Sun, 28 Aug 2016 21:13:57 -0400

> On Aug 28, 2016, at 7:10 PM, David Noveck <[email-masked]> wrote:
> 
> 
> There aren't any and I don't believe any are needed.  There is some text that says xid for updxch is not tied to that for the reqxch, but I think it need to be more clearly stated that there is no use for this.  I could either tell the receiver to ignore this or tell the sender to always make this zero.

Thinking aloud:

Generally speaking, I'm wondering about the rdma_xid field in any RPC-over-RDMA message
that does not carry an RPC message payload (and that would include any extension of
RPC-over-RDMA Version Two).

Seems to me that if there is no RPC message payload, that XID field could (even should?)
be used independently of the XID space used by the RPC client (the "upper layer").

In other words, a disjoint XID space could be used for non-RPC-message bearing messages
such as XCH operations. That field could be used for some other purpose entirely when
there's no need to provide call-reply transactional semantics.

We have the same problem if we were to, say, send more than one RPC message in a single
RPC-over-RDMA message.

--
Chuck Lever

---

From: karen deitke <[email-masked]>
Date: Mon, 29 Aug 2016 10:21:25 -0600

> problem for him doing it for  receive buffer size Y, with Y > X.
Okay, maybe thats a bad example.  I guess it would be something like 
disable remote invalidation?  Trying to understand what type of 
properties this might entail.
> > Is there any way for it to say I don't like that value, not now,
I see, makes sense for something like remote invalidation for example.

> that a REQXCH_RESPXCH pair, is, for purposes of credit management, to 
> be treated as if it were
>
Right, but that doesn't guarentee that a recv buffer will be posted.  
For example, on the solaris client, we post enough buffers for the 
#credits for backchannel request (if used for nfsv4.1, if 4.0 we won't 
post any because we expect tcp to be used), plus one for each 
outstanding request we send on the forechannel in which we are expecting 
a response.  Even though, for example, the total number of credits on 
the forechannel is 128.

So for example, if its v4.0, we will only have recv buffers posted for 
outstanding requests.  If the server wants to send a reqxch or updxch we 
won't have any buffers.  I suppose we could always post an additional 
one, but how does the server know how many we have?

> > believes the property can be changed, just maybe not to the > value 
> the sender wants, it needs to return the respxch with
> Actually the requested is not rejected.  The requested charcteristic 
> is included in optrespxch_pend nd there is a UPDXCH to follow up with 
> the selected value.
>
I see.

> There isn't one now but I am considering a change to UPDXCH in -03 
> that would include this.  The goal would not be to reduce the message 
> count, although it would do that.  Instead, the idea would to separate 
> solicited and unsolicited
I like that better.
> one xcharspec --> multiple xcharvals so changes in multiple properties 
> can be requested.
>
> > Also, why would an updxch be used at all, wouldn't the peer > just send its own reqxch?  Thats whats confusing 
> to me.  Is it > more like I'm not asking to change this propery, I'm 
> telling
> > you its changed?
>
Would there ever be a case where the sender of updxch needs to know that 
the receiver has received this before actually implementing the change 
on the sending side?

> Clearly the existing text isn't clear enough.  Do you have any 
> suggestions to make this clearer?
>
Examples  of why updxch is needed over simply using reqxch would be 
helpful.  Like maybe the characteristics of the property, for example, 
remote peer has no input on change, no reply needed upon receipt for 
sending to make change, etc.

  My other thought on updxch is, how does the side that wants to send 
this, know that the peer has a recv buffer posted for it?  For example 
in the solaris v4.0 case, there would not be any, unless we purposely 
post one and leave one posted. But then the sending can only send one 
updxch at a time.  If the receiver (i.e the nfs client) reposts a new 
recv buffer for the updxch after it receives this request, how does the 
sending side know that it had posted a new one and can now send another?

> There aren't any and I don't believe any are needed.  There is some 
> text that says xid for updxch is not tied to that for the reqxch, but 
> I think it need to be more clearly stated that there is no use for 
> this.  I could either tell the receiver to ignore this or tell the 
> sender to always make this zero.
>
Chuck's response on a different xid space is an interesting thought.
> the unsolicited change move things in the same or a different 
> direction than the change request.
>
Being the xid's are different, wondered if the reqxch sender would wait 
indefinitely for an repxch.  Hmm, can a given
property be changed by a reqxch or updxch? Or would the property be such 
that only updxch can change it if the receiver of updxch has no input?

> > Would it make sense with the optinfo_initxch to also have a > supported max/min value that would possible be 
> accepted if > a change were to be made?
>
> consider for -03. Let's leave this for -04 or after this is merged 
> into the version-two document.
>
> ordered linearly with a min and max.  For example,there here are 
> properties in rpcrdma-rtrext that don't follow that model.
>
Right, remote invalidation support I suppose, for example.

> > Is see that xcharspec is typedef'ed to xcharval and its really the 
> same thing,
>
Ah, I see, that makes much more sense now.
> I think it would help to rename xcharspec to be xcharvalset.
Yes, that would be helpful.

Karen
>
>
>
> On Fri, Aug 26, 2016 at 7:57 PM, karen deitke <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Does the receiver have any input on the unsolicited xchar update? 
>     low on memory.  Is there any way for it to say I don't like that
>     value, not now, rejected, etc?
>
>     support changes to properties, seems like they must at least have
>     1 additional recv buffer posted to accept the reqxch, is one enough?
>
>     If a reqxch were sent to make a change, and the receiver believes
>     the property can be changed, just maybe not to the value the
>     sender wants, it needs to return the respxch with rejected, and
>     then follow up with an updxch to increase the value possible just
>     a little bit more, just not to the value the sender wanted.  Seems
>     like maybe to avoid two messages, the receiver could have returned
>     a respxch but indicated that new value.  But I don't see a value
>     in the return in respxch.
>
>     If reqxch can only request 1 xcharspec, then why does respxch have
>     the ability to indicate more than one property were changed, or
>     rejected,etc?  Is this to allow multiple reqxch requests to be
>     notified of completion with 1 respxch?
>     If the respxch were one for one with the reqxch, seems we could
>     get rid of the concept of pending reqxch, especially if the
>     respxch had the value which could eliminate the updxch?  Or can
>     you put more than one reqxch in one messages?  That might be nice.
>
>     Also, why would an updxch be used at all, wouldn't the peer just
>     send its own reqxch?  Thats whats confusing to me.  Is it more
>     like I'm not asking to change this propery, I'm telling you its
>     changed?
>
>
>     Also, what guarentees that the xid of the updxch is unique among
>     the xid's the peer is using?
>
>     Would it make sense with the optinfo_initxch to also have a
>     supported max/min value that would possible be accepted if a
>     change were to be made?
>
>     Also seems that the updxch's optupdxch_now (new) should be
>     xcharspec.  Is see that xcharspec is typedef'ed to xcharval and
>     its really the same thing, but it makes for easier reading when
>     the reqxch is an xcharspec.
>
>
>
>     Karen
>
>
>
>
>>     It's not that it is not trusted.  It is that when you have a
>>     request outstanding to redeuce the the
>>     size to X, you need to take that account.  So, if you get an
>>     update that the size is now Y, you
>>     Yes, but in that case, the requester will find out that his
>>     request has been granted,
>>     work OK although people might have a problem with the
>>     administrator's choices.
>>
>>
>>
>>     I think I could draft something that would not run into the
>>     preclusion problem.
>>
>>     receive buffer size problmatic in the absence of a mechasim to
>>     nsure that the
>>     On Fri, Aug 26, 2016 at 11:39 AM, Chuck Lever
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>         Hiya Dave-
>>
>>         > On Aug 26, 2016, at 5:08 AM, David Noveck
>>         <[email-masked] <mailto:[email-masked]>> wrote:
>>         >
>>         > I'm OK with saying you shouldn't do that, but I have
>>         problem saying it is "acceptable".  I think it is noxious,
>>         although there is no "NOXIOUS".
>>         >
>     _______________________________________________ nfsv4 mailing list

---

From: David Noveck <[email-masked]>
Date: Tue, 30 Aug 2016 06:31:16 -0400

> So for example, if its v4.0, we will only have recv buffers
> posted for outstanding requests.  If the server wants to send > a reqxch
or updxch we won't have any buffers.

I suppose I could respond by saying nasty things about v4.0's
handling of callback but there a few problems with that:

   - It doesn't help solve the problem
   - It really is my fault :-(

> I suppose we could always post an additional one, but how
> does the server know how many we have?

Good point.

Let me talk about a likely approach to this issue for -03, but
I think this will not be fully resolved until the merger of xcharext and
version-two.   It appears that I haven't really been taking account of the
fact that backward-direction operation is OPTIONAL in version-two and that
in making REQXCH-RESPXCH a quasi-rpc, I am making some variants of it a
quasi-callback.

I think the way to deal with this is to state that the credit field  in the
INITXCH is a credit grant and if this is received as zero, the sender is
essentially asking the receiver not to send REQCH or UPDXCH (or any rpc
callbacks) in the future.  This is the same as not supporting these
OPTIONAL types but you could also send in the INITXCH a characteristic that
indicated that the sender (in this case the client) wouldn't support these
anyway, even if buffers were available.

On Mon, Aug 29, 2016 at 12:21 PM, karen deitke <[email-masked]>
wrote:

> disable remote invalidation?  Trying to understand what type of properties
> this might entail.
>
> Right, but that doesn't guarentee that a recv buffer will be posted.  For
> example, on the solaris client, we post enough buffers for the #credits for
> backchannel request (if used for nfsv4.1, if 4.0 we won't post any because
> we expect tcp to be used), plus one for each outstanding request we send on
> the forechannel in which we are expecting a response.  Even though, for
> example, the total number of credits on the forechannel is 128.
>
> won't have any buffers.  I suppose we could always post an additional one,
> but how does the server know how many we have?
>
> the receiver has received this before actually implementing the change on
> the sending side?
>
>  My other thought on updxch is, how does the side that wants to send this,
> know that the peer has a recv buffer posted for it?  For example in the
> solaris v4.0 case, there would not be any, unless we purposely post one and
> leave one posted. But then the sending can only send one updxch at a time.
> If the receiver (i.e the nfs client) reposts a new recv buffer for the
> updxch after it receives this request, how does the sending side know that
> it had posted a new one and can now send another?
>
>> On Fri, Aug 26, 2016 at 11:39 AM, Chuck Lever < <[email-masked]>
>> [email-masked]> wrote:
>>
>>> Hiya Dave-
>>>
>> [email-masked] https://www.ietf.org/mailman/listinfo/nfsv4
>
>