Subject: [Tsvwg] RDMA over IP Requirements draft

From: "Talpey, Thomas" <[email-masked]>
Date: Fri, 17 May 2002 09:24:27 -0400

A few of us have authored an initial set of requirements
for RDMA over IP, which we've published as an Internet
Draft. We've also included a terminology section, to anchor
a common set of terms for the effort moving forward.

Discussion is welcome on the RDMA over IP mailing list at
 	http://groups.yahoo.com/group/rdma/

Tom Talpey, NetApp
David Robinson, Sun
Robert Teisberg, HP
Jim Wendt, HP

-----

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: RDMA over IP (ROI) Requirements
	Author(s)	: T. Talpey et al.
	Filename	: draft-talpey-rdma-over-ip-requirements-00.txt
	Pages		: 22
	Date		: 15-May-02
	
This draft defines terminology and requirements to be used in
conjunction with the RDMA over IP (ROI) effort.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-talpey-rdma-over-ip-requirements-00.txt

To remove yourself from the IETF Announcement list, send a message to 
ietf-announce-request with the word unsubscribe in the body of the message.

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-talpey-rdma-over-ip-requirements-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-talpey-rdma-over-ip-requirements-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 17 May 2002 10:46:08 -0700

Thomas,

What is implied by the vertical bars within ULP and Transport?  Is this to
imply more than one?  If so, it would be a clearer diagram if these bars
were removed.

 +-------------------+--------------+----------------+
 |       ULP         |     ULP      |      ULP       |
 +-----+-------------+-------------------------------+
 |     |             |             RDMA              |
 |     |             +-------------------------------+
 |     |                     DDP                     |
 |     +--------------------+------------------------+
 |        Transport         |      Transport         |
 +--------------------------+------------------------+

May I suggest that the terms packet or segment be changed to clarify these
terms are not always synonymous with terms used by the transport.  Perhaps
EPDU for Encapsulated PDU could serve as a substitute for segment.  I
realize the goal would be to make these terms synonymous as is possible for
some transports.

Data handling is not always composed of simple blocks and, as these
structures are affected by architecture, constructs used by DCE-RPC/IDL or
ONC-RPC/XDR could be considered an efficient alternatives to decomposition
into separate DDP statements for each element within a structure.  DDP would
still be effective in allowing direct placement of IDL defined structures
and could be completely transparent to present applications.  If this IDL
feature is not supported, then DDP decomposition for each element would be a
less effective fall-back mode.

As present cross-platform standardization relies on structure conformance,
transforming elements 2, 4, and 8 bytes at a time can be done 'on the fly'
within the same hardware that does DDP 'on the fly'.  Template exchanges and
references as now done for environments based on DCE-RPC/IDL such as ActiveX
and .Net suggests this approach is already endemic.  Would it be appropriate
to look at this level of overhead detail?  The present SCTP-DDP draft
supports this mode of operation.

 -Doug

> A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-talpey-rdma-over-ip-requirements-0
0.txt

Send a message to:
	[email-masked].
In the body type:

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 17 May 2002 12:16:43 -0700

David,

See comments interspersed -

On May 17, 2002 11:18 AM David Robinson ([email-masked]) wrote:
>
> Douglas Otis wrote:
>
> > What is implied by the vertical bars within ULP and Transport?
> > Is this to imply more than one?  If so, it would be a clearer
> > diagram if these bars were removed.
>
> It implies both multiple ULPs and multiple Transports as well
> as different ULPs will be layered directly on any one of the
> layers. I don't see how it could be clearer, what specifically
> would see it conveying differently?

As a diagram, the generic term ULP and Transport implies plural.

> > May I suggest that the terms packet or segment be changed to
> > clarify these terms are not always synonymous with terms used
> > by the transport.  Perhaps EPDU for Encapsulated PDU could
> > serve as a substitute for segment.  I realize the goal would
> > be to make these terms synonymous as is possible for some
> > transports.
>
> I agree that packet should be avoided unless we are specifically
> talking about the bits on a wire and it is true that the segment
> we are using is not necessarily the same segment as a transport
> would use.  But I am personally not a fan of EPDU as it seems to
> imply more semantics than is necessary, but I am interested in
> alternatives to segment.

Okay, at least we agree this term should change.

> > Data handling is not always composed of simple blocks and, as
> > these structures are affected by architecture, constructs used
> > by DCE-RPC/IDL or ONC-RPC/XDR could be considered an efficient
> > alternatives to decomposition into separate DDP statements for
> > each element within a structure.  DDP would still be effective
> > in allowing direct placement of IDL defined structures and
> > could be completely transparent to present applications.  If
> > this IDL feature is not supported, then DDP decomposition for
> > each element would be a less effective fall-back mode.
> >
> > As present cross-platform standardization relies on structure
> > conformance, transforming elements 2, 4, and 8 bytes at a time
> > can be done 'on the fly' within the same hardware that does DDP
> > 'on the fly'.  Template exchanges and references as now done for
> > environments based on DCE-RPC/IDL such as ActiveX and .Net
> > suggests this approach is already endemic.  Would it be
> > appropriate to look at this level of overhead detail?  The
> > present SCTP-DDP draft supports this mode of operation.
>
> I am not convinced that breaking down an IDL or XDR encoding into a
> DDP per field or construct is necessary. The overhead and benefit of
> such small messages just doesn't seem justified.  Such structures are
> usually just a small fraction of data flow, at best I could see two
> DDP messages, one for the headers and one for the data.  As an
> example, a typical NFSv3 read response is ~128 bytes of header and
> 32KB of data, it is clearly advantagous to put the 32KB data in a DDP
> message and may be some benefit to putting the headers in another DDP
> message, but I can't see much use for breaking up things further.

Headers for DFS, AFS, CIFS, DAFS, and NFS would be more effective if these
structures could be exchanged as appropriate for the recipient architecture.
There are also many applications that exchange structures and not blocks.
ADO.Net and ODBC would be examples of information that could be dubbed as
structure rich.  If the only goal was RDMA, then this consideration easily
gets lost.  If the goal is to support effective network use for a broad
range of applications, then structure exchange should be an aspect that
receives more consideration.  I agree this would be more relevant to DDP
than RDMA.  If RDMA is built upon DDP, then effective header utilization by
the receiver enables efficient RDMA requests as an extension to DDP and thus
makes it relevant.

> In terms of requirements, I don't think there is anything in the
> draft that would prohibit creating such fine grained DDP messages. It
> may not be the most efficient use of DDP but it is allowed.  Is there
> another, or clearer, requirement that you envision?

I'll continue review, but it looks like a good start.

 -Doug

> 	-David

---

From: David Robinson <[email-masked]>
Date: Fri, 17 May 2002 15:22:58 -0500

> structures could be exchanged as appropriate for the recipient architecture.

Are you advocating yet another mechanism to specify byte ordering? About
every possibly permutation has been done over the years (cannonical
network format, sender makes it right, receiver makes it right) but
in hindsight, despite all the arguments about which is "better", history
has shown it not to be a problem. Modern processors (like SPARC, IA-64)
have unprivleged processor bits that allow you to control the byte
order in loads and stores, so performance is no longer an issue.

> the receiver enables efficient RDMA requests as an extension to DDP and thus

While I agree to some extent that networking protocols are increasingly
becoming much more structured (XML, WebDAV, etc) and that is providing
much richer capabilities, I am extremely hesitant to allow this group
to branch outside if its narrow charter, one that some still think is
too broad, and into the mire of data description languages and
encoding schemes.  A historically very controversial area. We have
enough pits that we can't seem to get out of that I don't want to
create new and deeper ones.

I would assert that: the ROI protocols treat the ULP payloads as
uninterpreted and unmodifiable byte streams.

To do anything different will require a change in charter, and
will likely send us outside the transport area and into
applications.

	-David

---

From: David Robinson <[email-masked]>
Date: Fri, 17 May 2002 13:17:45 -0500

It implies both multiple ULPs and multiple Transports as well
as different ULPs will be layered directly on any one
of the layers. I don't see how it could be clearer, what
specifically would see it conveying differently?

> supports this mode of operation.

I am not convinced that breaking down an IDL or XDR encoding into a DDP
per field or construct is necessary. The overhead and benefit of such
small messages just doesn't seem justified.  Such structures are
usually just a small fraction of data flow, at best I could see two DDP
messages, one for the headers and one for the data.  As an example,
a typical NFSv3 read response is ~128 bytes of header and 32KB of data,
it is clearly advantagous to put the 32KB data in a DDP message and
may be some benefit to putting the headers in another DDP message,
but I can't see much use for breaking up things further.

In terms of requirements, I don't think there is anything in the draft
that would prohibit creating such fine grained DDP messages. It may not
be the most efficient use of DDP but it is allowed.  Is there
another, or clearer, requirement that you envision?

	-David

---

From: "Douglas Otis" <[email-masked]>
Date: Fri, 17 May 2002 18:09:34 -0700

David,

While I agree that DDP should treat payloads as opaque objects, applications
wishing distributed processing of data demands structures be defined and
converted.  There has been a considerable amount of consolidation in this
area with regard to interface structure definitions.  As this relates to DDP
or ROI, providing a means to define or imply these definitions is a minor
effort but would help minimize hardware needed.

This could relate to the use of PPI within SCTP-DDP, as example.  This can
be expanded upon by the ULP and perhaps even enhanced by hardware
conversion.  Either way, this is intended as an 'on-the-fly' process.
Regardless of processor capabilities, uniform data structures is still a
*vital* requirement for transparent use.  There is a vast heritage in this
area that should not be ignored.  Structures above DDP should comply with
the use of an IDL (Interface Definition Language).

This does not imply yet another byte-order specification.  In '55, John
Backus of IBM announced Fortran in which Backus developed a syntax for
defining terms.  This became known as BNF (Backus Normal Form). In '60,
Peter Naur of IBM announced Algol in which Naur modified the Backus Normal
Form.  E. W. Dijkistra, suggested keeping the initials and simply change the
"N" from Normal to Naur.  Impact is seen with variable size elements.
Creating formats for requests above DDP should consider use of an IDL.
Again, this should represent a minor impact.

I do not understand how this would relate to your examples, so I suspect we
are not discussing the same thing.  Much of this has become hidden as a
basic network function and is often forgotten as being an essential element
in distributed processing.  It is understandable that focus upon storage and
RDMA would ignore this aspect however.  I think it deserves more
consideration during the development of request structures.  This IDL
consideration allows extensibility and will minimize hardware needed.

 -Doug

On May 17, 2002 1:23 PM David Robinson ([email-masked]) wrote:
>
> Douglas Otis wrote:
>
> > Headers for DFS, AFS, CIFS, DAFS, and NFS would be more
> > effective if these structures could be exchanged as
> > appropriate for the recipient architecture.
>
> Are you advocating yet another mechanism to specify byte ordering?
> About every possibly permutation has been done over the years
> (cannonical network format, sender makes it right, receiver makes it
> right) but in hindsight, despite all the arguments about which is
> "better", history has shown it not to be a problem. Modern processors
> (like SPARC, IA-64) have unprivleged processor bits that allow you to
> control the byte order in loads and stores, so performance is no
> longer an issue.
>
> > There are also many applications that exchange structures and
> > not blocks. ADO.Net and ODBC would be examples of information that
> > could be dubbed as structure rich.  If the only goal was RDMA, then
> > this consideration easily gets lost.  If the goal is to support
> > effective network use for a broad range of applications, then
> > structure exchange should be an aspect that receives more
> > consideration.  I agree this would be more relevant to DDP
> > than RDMA.  If RDMA is built upon DDP, then effective header
> > utilization by the receiver enables efficient RDMA requests as an
> > extension to DDP and thus makes it relevant.

>
> While I agree to some extent that networking protocols are
> increasingly becoming much more structured (XML, WebDAV, etc) and
> that is providing much richer capabilities, I am extremely hesitant
> to allow this group to branch outside if its narrow charter, one that
> some still think is too broad, and into the mire of data description
> languages and encoding schemes.  A historically very controversial
> area. We have enough pits that we can't seem to get out of that I
> don't want to create new and deeper ones.
>