Subject: [rddp] Rgerading RDMA/SDP and traditional socket apps

From: Mohan Parthasarathy <[email-masked]>
Date: Fri, 01 Oct 2004 10:28:58 +0530

Hi,

  I am new to RDMA and would appreciate if my questions are viewed
  with tolerance.
  I believe existing socket apps doing a send and reusing the buffer will
  not work with RDMA since the send might not have completed.
  Isn't it possible to copy the user buffer to another in the SDP library,
  to let the user app reuse the buffer ? I guess it might lead to increase
  in memory usage, but it will be only for legacy apps not using the
  RDMA capabilities, and will probbaly be no different from today's
  situation where often the user buffer gets copied to a kernel buffer.
  Also, are there any new socket APIs in SDP or variations in the current
  socket calls to distinguish if the caller can handle running over RDMA ?
  Another way to phrase that is if SDP will be aware of legacy apps
  versus new ones.

Regards,
Mohan

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 1 Oct 2004 11:04:07 +0200

>  Another way to phrase that is if SDP will be aware of legacy apps
The purpose of SDP is to identify when zero-copy transfers
can be done, directly using either the ULP source or destination
buffer directly, without requiring a fundamental change in the
how the application uses the network.

Without SDP a stack can only identify zero-copy opportunities
locally, without collaborating with the peer's stack.  However,
an optimized local stack can identify many zero-copy opportunities,
and when it does it is without the overhead and latency of
additional  wire traffic.  The tradeoff between local-stack-only
optimization and SDP-enabled optimization is complex
and dependent on the application and deployment environment.

Obviously an RDMA aware application that explicitly posts
receive operations and explicitly exports STags to advertise
buffers is going to avoid intermediate copying *all* of the
time, something that SDP can only hope to approach.

Both an optimized local stack, such as a TOE, and SDP
are valuable tools to achieve partial zero-copy without
requiring program redesign.  Achieving the full potential
of RDMA requires designing the Upper Layer Protocol
and its implementation on both peers to be RDMA aware.

The SDP specification is an excellent source on when and
how SDP identifies zero-copy opportunities. The Applicability
Statement discusses the impact of RDMA on application design.

---

From: Michael Krause <[email-masked]>
Date: Fri, 01 Oct 2004 07:12:11 -0700

>  Another way to phrase that is if SDP will be aware of legacy apps

If the sockets call returns, then the ownership of the associated buffer is 
also returned.  SDP does not change this.  Therefore, the application can 
reuse the buffer.  Today's network stacks do not guarantee that a buffer 
has been sent when a sockets call returns.  All that is guaranteed is that 
the underlying implementation will attempt to deliver the buffer to the 
remote.  The same applies to RDMA implementations.  The implementation will 
attempt to deliver the buffer.  If the buffer is returned, then either it 
was successfully sent or there is a local copy within the RNIC or a side 
buffer (e.g. a local copy to host memory) that will be used to send the 
data.  The application does not need to worry about this.  SDP does not 
have to worry about this either.  This is the RI / RNIC's 
responsibility.  A good explanation can be found in the verbs draft 
specification which has been posted to this reflector / website.

Mike

---

From: Mohan Parthasarathy <[email-masked]>
Date: Mon, 04 Oct 2004 08:34:52 +0530

Hi Mike/Caitlin,

  Thanks for the answers. So, what I understand is that my assumption
  that existing socket apps might not have ownership of the buffer when
  the send completes, was wrong.
  But, doesn't this mean that send calls will take more time to complete,
  if the RNIC has to DMA the buffers before control to the buffer is
  returned to the caller. Are there any potential performance issues here
  for existing synchronous sockets ?

Regards,
Mohan

Michael Krause wrote:

> If the sockets call returns, then the ownership of the associated 
> buffer is also returned.  SDP does not change this.  Therefore, the 
> application can reuse the buffer.  Today's network stacks do not 
> guarantee that a buffer has been sent when a sockets call returns.  
> All that is guaranteed is that the underlying implementation will 
> attempt to deliver the buffer to the remote.  The same applies to RDMA 
> implementations.  The implementation will attempt to deliver the 
> buffer.  If the buffer is returned, then either it was successfully 
> sent or there is a local copy within the RNIC or a side buffer (e.g. a 
> local copy to host memory) that will be used to send the data.  The 
> application does not need to worry about this.  SDP does not have to 
> worry about this either.  This is the RI / RNIC's responsibility.  A 
> good explanation can be found in the verbs draft specification which 
> has been posted to this reflector / website.
>
> Mike

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 4 Oct 2004 06:46:10 +0200

>    for existing synchronous sockets ?

SDP defines multiple transfer procedures precisely to avoid
this problem.  SDP should be configured so that zero-copy
transfers are only done when doing so will improve performance.

For example,  when the receive  is posted long enough before
the send for the data sink buffer to have been advertised the
data source can RDMA Write to the final target buffer without
any delay.

SDP inherently involves more wire traffic, but depending on
the timing of sends and receives this extra traffic may have no
impact on latency.  The benefits of direct placement should
overcome any penalty for the wire traffic.  But that penalty is
there, which is why local-only optimizations that achieve direct
placement without wire traffic might be more efficient, and why
full adaptation to RDMA-aware application logic will almost
always be more efficient.

---

From: Vadim Makhervaks <[email-masked]>
Date: Mon, 4 Oct 2004 14:13:37 +0200

Caitlin,

I tend to agree with Mohan's observation. Preposted receive buffers allow
to eliminate a need in buffer advertising, but it still involves delay. I
can see two options: option a) RNIC does not keep any intermediate buffer,
and then it needs to wait to Ack to arrive, before releasing consumer
buffer, option b) RNIC keeps intermediate buffers, and then consumer
buffer is released after it is copied to the intermediate buffer. In both
cases we have delay involved in blocking send operation, plus some sort of
hw/sw sync/coordination to report completion of operation and release a
buffer.

Vadim

|---------+---------------------------->
|         |           Caitlin Bestler  |
|         |           <[email-masked]> |
|         |           Sent by:         |
|         |           rddp-bounces@ietf|
|         |           .org             |
|         |                            |
|         |                            |
|         |           04/10/04 06:46   |
|---------+---------------------------->
  >-------------------------------------------------------------------------------------------------------------------------------|
  |                                                                                                                               |
  |       To:       RDDP <[email-masked]>                                                                                          |
  |       cc:       Mohan Parthasarathy <[email-masked]>                                                                     |
  |       Subject:  Re: [rddp] Rgerading RDMA/SDP and traditional socket apps                                                     |
  >-------------------------------------------------------------------------------------------------------------------------------|

---

From: Sukanta ganguly <[email-masked]>
Date: Mon, 4 Oct 2004 06:47:01 -0700 (PDT)

Hi,
  I do not think the issue that Mohan is raising is
about more traffic across the wire. In essence the
traffic on the wire will most probably be less that
than the none SDP protocol is concerned.
   The question of whether the application with
synchronous socket approch will have to wait longer
for the RNIC to do the DMA onto its local buffer
before the app on the other side get controll of its
buffer is concerned is a valid one.
   So the zero copy effect is one aspect that will
reduce the amount of real time the synchronous socket
I/O app will face latency issues. As far as the reuse
of the buffer is concerned, it will depend on a lot of
elements, the LAN/WAN link speed will be one, the
performance of the remote node will be another one.
   So o clearly answer this the question, the
application should be designed with more memory to be
used as during such communication process it may not
have the reusebility of its buffers at a very high
rate. I think higher memroy consumption will be an
element to wonder rather than latency, though latency
is also an after effect.

Thanks
SG

--- Caitlin Bestler <[email-masked]> wrote:

> >    that existing socket apps might not have
> ownership of the buffer 
> > when
> >    But, doesn't this mean that send calls will
> take more time to 
> > complete,
> >    returned to the caller. Are there any potential
> performance issues 
> > here

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 4 Oct 2004 09:52:15 -0500 (CDT)

> element to wonder rather than latency, though latency

Good point.

But it is important to remember that this can be more of an
accounting issue than an actual savings of memory.

Use of SDP, or for that matter an RDMA-aware interface,
will have the effect of replacing usage of system buffers
with usage of application buffers.

The benefit of zero-copying may result in resulting in
more total buffering, or it might merely be transferring
the same bytes from column A (system buffering) to
column B (application buffering).

The applicability statement covers these issues, though
mostly concentrating on receive buffering.  There are
applications that truly need the benefits of shared buffering
across multiple applications -- and they might be poor
candidates for RDMA.  For many other applications, and
I personally believe most, the system buffer is a bottleneck
rather than an enabler.  It is much easier to double the
amount of RAM in your system than to double its bandwidth.

---

From: Michael Krause <[email-masked]>
Date: Mon, 04 Oct 2004 09:22:29 -0700

Some things to keep in mind:

- Synchronous sockets calls can be completed without any data actually 
being sent in a traditional network stack.  Kernel buffers the data to 
transmit / re-transmit and the application proceeds normally.  This model 
does not reduce total memory consumption.  As link speeds increase, 
synchronous call will either fail (if non-block set) or stall until kernel 
memory becomes available as a function of acknowledgement rate.

- Asynchronous sockets calls can be completed without any data being 
sent.  This is no different than synchronous sockets calls.  However, 
asynchronous can use the application buffer for retransmission (assuming 
operation is not completed to the consumer) thus reducing memory pressure 
in the kernel.

- Asynchronous calls (e.g. sockets) map best to RDMA infrastructures and 
provide optimal performance as well as memory resource management.

- Synchronous sockets calls can be supported with SDP and can use the 
one-copy or zcopy modes to mitigate local as well as end-to-end application 
latency.

- RDMA infrastructure enables OS bypass operation and the ability to 
mitigate interrupts within the system.  This can yield a much more 
responsive solution as well as more predictable behavior as system load 
increases.

- If applications use synchronous sockets, then they should be designed to 
deal with the inevitable stall due to a blocked system call.  Use of SDP is 
orthogonal to this and therefore its presence should not have an impact on 
application design.  Yes, programmers often make assumptions about how fast 
a call will return but if they are creating a robust design, then they 
should have accounted for the stall..

Note: The OpenGroup Interconnect Software Consortium is defining a set of 
Sockets extensions that provide asynchronous interfaces and explicit memory 
management.  These extensions will enable Sockets applications to fully 
exploit the performance benefits of RDMA while enabling the OS to take the 
guess work out of memory management.  These extensions are to be part of 
the Unix branding so should be available across all Unix and hopefully 
Linux implementations in the next couple of years.  The specification is 
nearly complete and should be final by the end of this year.

Also note that Winsocks has been shipping async sockets support for quite 
some time and that Winsocks Direct, which is the basis for SDP, is a good 
proof of concept of the performance benefits.  There are a couple of white 
papers that provide a good overview of what one might expect when running SDP.

Mike