Subject: Re: [nfsv4] RPC/RDMA Stuff

From: Chuck Lever <[email-masked]>
Date: Mon, 19 Oct 2015 15:49:43 -0400

I will attempt to address your other comments by revising
the -02 submitted today. Meanwhile, there are a couple of
items that need to be discussed further on-list.

> On Oct 19, 2015, at 1:50 PM, David Noveck <[email-masked]> wrote:
> 
> 2.5.3 Read Chunk
> 
> I see some problems here:
> 	• First,, I found the use of the term "argument" confusing.  While the data to be written by an NFSv3 WRITE is clearly an argument of the WRITE RPC, the situation is different in NFSv4.  The buffer to be written by the WRITE operation is not an argument of the COMPOUND RPC.    It is part of the argument but is not an argument itself.

This is problematic because RFC 5666 doesn’t make clear that
the purpose of a chunk is to capture a single opaque XDR
structure, and not simply an arbitrary group of bytes in the
XDR stream. This organization makes it straightforward to
integrate RDMA into existing XDR implementations.

The example of NFSv4 COMPOUND is troubling. The intention of
both RFC 5666 and 5667 is that a WRITE operation in an NFSv4
COMPOUND should work just the same way as an NFSv3 WRITE
operation. The opaque data payload argument, without its
length, is captured in a read chunk. Nothing more. I would
like to improve the language in bis to make this clear, and
am open to suggestions.

> 	• The possibility that RPC object can be nested within one another, means that the same XDR position can be associated with multiple RPC objects that start at the same position and end in different positions.  This possibility needs to be addressed in some way.

The same XDR Position cannot be associated with multiple XDR
objects. I’m not sure I understand what you mean by “start at
the same position but end in different positions”. A list of
read segments can have the same value in their XDR Position
field, and that means precisely that they are concatenated,
in list order, by the receiver, into the sink buffer.

This is discussed in “Read Chunk”, and can be amplified in
the “Read List” section.

> 	• In cases in which you have a counted array, the XDR position of the counted array is the first byte of the length.  on the other hand, for practical reasons, you want don't want to include the length in the area you RDMA.  It has to be made clear how this issue is dealt with.

There are two interesting examples of a counted array:

1. A counted array of bytes. In this case, the length is not
included in the chunk.

2. A counted array of complex XDR objects. I believe the array
length is also not included in the chunk in this case. However
I would argue that complex XDR objects need to be marshaled,
and therefore would never be eligible for RDMA (except when
conveyed via a long message).

RFC 5666 already states that the length of an XDR object is
not included in the chunk. I assumed this applied to arrays.

> 2.5.4 Write Chunk
> 
> It is not made clear here (or in 2.5.6) how the appropriate result/object is selected, given that the write chunk doesn't have an XDR position.  I assume that successive list entries are assigned to successive RDMA-eligible result objects as defined by the ULB, but that needs to be stated explicitly.

Good point, and your assumption that the ULB plays a key role
here is correct, because there is no Position field in write
chunks.

> Also the ULB gets to decide where in the object the RDMA-eligible piece is (i.e. it need to address the write-chunk version of the counted array issue).

Again, a counted array is exactly what we have with an NFS
READ payload, and the array length is transferred inline
while the payload bytes are transferred in a write chunk.

If you still believe the I-D is inadequate, please do 
suggestion some edits, and I will consider them.

> 3.7 Write List Error Cases
> 
> It isn't clear what sort of situation is being referred to in the penultimate paragraph.  If the ULB defines what is RDMA-eligible, I don't see how the server can have any "choice" as to what to transfer via RDMA.

Each NFSv4 COMPOUND is an array of switched unions.

If there is more than one operation in a compound that has
a union result with arms that may include an RDMA-eligible
XDR object, and arms that do not include one, some way will
need to be found to sort out which write chunk goes with
which result.

Multiple READ_PLUS operations in a compound is the case I’m
thinking of.

--
Chuck Lever
[email-masked]