Subject: RE: [rddp] Connection Startup

From: "Kanevsky, Arkady" <[email-masked]>
Date: Fri, 3 Oct 2003 14:05:24 -0700

> From: Jim Pinkerton [mailto:[email-masked]] 
> First, I agree that for DAT (which was defined for 
> InfiniBand, as well as older, proprietary MAC layer 
> encapsulations of RDMA) there is a need, at least as DAT is 
> currently defined.

There is a requirement for current applications using existing RDMA
transport technology
to rely on the RDMA transport provided connection establishment.
Some applications can define their own protocol for connection
establishment.
SDP is an example of it.

DAT quotation is a red hearing since it is just an API and not a
protocol or an appication.

Arkady

Arkady Kanevsky                                email: [email-masked]
Network Appliance                              phone: 781-768-5395
375 Totten Pond Rd.                            Fax: 781-895-1195
3rd Floor                                      http: www.netapp.com
Waltham, MA 02451-2010                         general phone:
781-768-5300

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Jim Pinkerton" <[email-masked]>
Date: Fri, 3 Oct 2003 14:38:46 -0700

I agree with your conclusion that we both support the capability. 

I think we still disagree on the use of the capability, let alone who
will use it. Specifically I don't see how the capability can solve
transport layer independence at all. Many of the issues must be dealt
with before the connection is even setup (like will the LLP provide
datagrams or streams, or does the transport provide reliability, etc).
The handshake doesn't address those issues. It does enable a middleware
library to attempt to address them, and push through some private data
to try to set things up between itself and its peer at the remote end.

Jim

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, October 03, 2003 1:38 PM
> To: Jim Pinkerton
> Cc: [email-masked]
> Subject: Re: [rddp] Connection Startup
> 
> 
> On Friday, October 3, 2003, at 03:16 PM, Jim Pinkerton wrote:
> 
> > Note that iSER essentially uses the same boot start mechanism - send
a
> > bit of data, agree on configuration, and then enable RDMA - the only
> > difference is that iSER exchanged the information as part of *their*
> > record delineation mechanism, and this proposal defines it within
MPA
> > (i.e. LLP) to make it possible to delineate config information
record
> > boundary independent of application protocol.
> 
> 
> But iSCSI negotiations are only defined over TCP, hardly LLP neutral.
> 
> When/if iSCSI/SCTP is defined (as it should have been) the ULP would
> have to engage in different interactions to negotiate depending on
> which LLP was in use. The bottom line is that TCP exchanges bytes
> while SCTP exchanges messages.
> 
> If you are developing an RDMA specific application then the message
> orientation is closer to how you will be working once RDMA is
> established.
> So having a message-oriented negotiation standard available as an
> option is a major benefit. It lets the entire application be coded
> at a "message" layer, not mixing byte streams and messages.
> 
> It's really early in the process to come to any final conclusions
> about what is "normal" for RDMA applications. The app developers
> that I have talked with have had no interest in adding iSER like
> capabilities to the DAT API.
> 
> But in any case, I believe we are in full agreement that *both*
> models need to be supported. Which just disagree over which one
> should be viewed as "normal".
> 
> 
> Caitlin Bestler - [email-masked] - http://asomi.com/
>

---

From: "Jim Pinkerton" <[email-masked]>
Date: Fri, 3 Oct 2003 16:32:13 -0700

I'm not sure we're going to converge on this discussion, so I suggest we
agree to bury the hatchet, make sure the feature works, mandatory to
implement, and optional to use, and move on.

Jim

> Sent: Friday, October 03, 2003 3:34 PM
> To: Jim Pinkerton
> Cc: [email-masked]
> On Friday, October 3, 2003, at 04:48 PM, Jim Pinkerton wrote:
> 
> > The reason I bring this up so strongly is for *new* protocols, which
> > are
> > designed to run on top of IWARP, and don't care about compatiblity
with
> > InfiniBand, I see no need for this capability. It's far more optimal
to
> > embed their config data in their existing configuration handshakes,
and
> > not add a new round-trip handshake.
> 
> That is a valid concern when adding an RDMA mode to an *existing*
> protocol, such as iSCSI. And the number of *existing* protocols
> that are LLP neutral is, well, pretty much zero.
> 
> Use of an RDMA negotiation message *after* other negotiations would
> indeed be a wasteful additional round-trip. The intent is to support
> applications that only need minimal simple parameters exchanges in
> order to establish the session. When this topic has been discussed
> in the DAT Collaborative, there has been little interest in engaging
> in more complex session setup.
> 
> The only thing that *must* be negotiated before enabling RDMA is
> the essential RDMA configuration (identifying any existing session,
> RDMA read credits, maximum untagged message size, etc.) to the
> extent that it is not pre-dictated as constants for that ULP.
> 
> The general conclusion I have seen application developers who
> are working on an RDMA specific applications is that complex
> detailed negotiations can be done *after* the connection is
> established.
> 
> Now you can argue that they were merely thinking that way
> because that is the only model that InfiniBand supported.
> Or perhaps, that is the only model that InfiniBand
> supported because it is a fine model that meets the
> needs of ULP designers who are assuming RDMA capability.
>

---

From: "Jim Pinkerton" <[email-masked]>
Date: Fri, 3 Oct 2003 18:25:36 -0700

One technical clarification on a comment below. You state:

> Both DAFS and NFS over iWarp match the private data
> approach. iSER is the only protocol proposed which negotiates in
> streaming mode and then shifts to RDMA mode.

I'm still coming up to speed on the NFS proposal, but it looks like it
negotiates in streaming mode, and upgrades the connection into RDMA
Mode. See section 2.5.3 in
http://www.ietf.org/internet-drafts/draft-talpey-nfsv4-rdma-sess-00.txt.

I know at least a one of the NFS over RDMA authors monitors this alias -
is the above statement correct?

Jim

> Sent: Thursday, October 02, 2003 7:59 AM
> To: Jim Pinkerton
> Cc: [email-masked]; David Black
> On Wednesday, October 1, 2003, at 10:36 PM, Jim Pinkerton wrote:
> 
> >
> > I don't see any real harm in an optional negotiation frame, since it
is
> > not required. But I think it is interesting to point out that the
> > example below makes it obvious that any content in the optional
> > negotiation frame is application specific (i.e. we are not
attempting
> > to
> > standardize (nor should we) what the contents of the start frame
are).
> >
> 
> Yes, the intent is to carry opaque messages between the ULP peers
> to enable them to select/configure the RDMA endpoints compatibly.
> 
> > For InfiniBand, there are a bunch of different definitions for the
> > private data contained in the connection setup sequence, and each
one
> > is
> > application dependent. There are a few parameters that are common,
but
> > the reality is little else is (see the InfiniBand spec Appendices at
> > http://infinibandta.org for the details).
> >
> > Thus to me the only rationale for creating the optional negotiation
> > frame is to enable a 3rd party to strip off the application data
from
> > the data stream, and present it through a separate interface to the
> > application. This is what Caitlin outlined below. From my jaded
view,
> > this is essentially "how do you graft software infrastructure that
was
> > created for InfiniBand on top of iWARP?".
> >
> 
> If that were the goal, the "private data" would have included some of
> the standard fields for the CM protocol that are relevant --
especially
> those related to RDMA Read credits.
> 
> I believe a better statement of the goal is to encourage development
> of RDMA-aware transport neutral applications. For IETF purposes the
> critical transport neutrality is between SCTP and TCP.  The next
> draft of the applicability statement will expand on this (it is
> under co-author review now, but if you read the latest SCTP and
> MPA drafts you'll see that transport neutral RDMA connection setup
> is easily achievable). Being compatible with the IT-API,
> the DAT APIs and the InfiniBand CM is just a bonus.
> 
> There are indeed some applications that will be forced to deal with
> both the socket layer and the RDMA interface. But I believe this will
> be the exception. Both DAFS and NFS over iWarp match the private data
> approach. iSER is the only protocol proposed which negotiates in
> streaming mode and then shifts to RDMA mode.
> 
> One of the main benefits of RDMA is that it allows the ULP to work
> at a much higher level of abstraction. A standard exchange of
> messages *before* RDMA endpoints are selected/configured that
> can be done *without* knowing which LLP is in use is totally
> consistent with that goal.
> 
> It also *allows* the negotiation frame to be implemented by
> hardware/firmware/drivers and/or middle-ware. This is also
> a major benefit, because it *can* minimize context switching.
> 
> >
> >
> >

---

From: "Talpey, Thomas" <[email-masked]>
Date: Fri, 3 Oct 2003 18:47:15 -0700

A draft is only a proposal :-) Seriously, I can see NFS/RDMA going either
way, though you are correct the negotiation as proposed negotiates and
does not, in fact, require private data. I do not believe NFS/RDMA is yet
ready to say it never will. The DAFS protocol does use it.

I believe the presence of optional private data is a good thing, will increase
the applicability of DDP/RDMAP, and should be supported. I would like to
see it supported when connecting over SCTP as well, in fact.

Tom.