Subject: RE: [rddp] Re: Invalidation

From: [email-masked]
Date: Fri, 3 Oct 2003 20:07:20 -0400

> From reading your email, it sounds like you are thinking there is a
> layering violation? Is that your point?

No.  It appears that invalidation is a rather important mechanism
for actually using STags, and so, aside from historical reasons, what's
the rationale for it being an RDMAP operation rather than a DDP operation?

Another way to ask this is: does absence of STag invalidation from DDP
significantly reduce the usefulness of DDP without RDMAP?  Alternatively,
is invalidation sufficiently ULP (wrt DDP, e.g., RDMAP)-specific that
it needs to be left to DDP's ULPs to implement and customize in their
own fashion?

If yes, the next question will be why it belongs in the RDMAP layer,
as opposed to RDMAP providing local access to the DDP interface for
STag invalidation, and letting ULPs figure out how to
communicate STags when remote invalidation is in order.

I don't have a preconceived notion of the answers, but there's some
design rationale that needs to be brought out and discussed here.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 3 Oct 2003 20:02:41 -0500

> design rationale that needs to be brought out and discussed here.

As currently drafted, the DDP layer merely places the data from
valid DDP Segments, notes that the DDP Segment was placed,
and optionally notifies the next layer of completions.

STag invalidation must take place at or after completion
time, not at placement time. Therefore it would be awkward
to handle it within the DDP layer.

One of the justifications for the layering in my mind is
that it allows a very simple state machine to implement
the DDP layer, possibly leaving the RDMAP layer to a
more complex device (such as a processor).

Caitlin Bestler - [email-masked] - http://asomi.com/

---

From: "Mallikarjun C." <[email-masked]>
Date: Fri, 3 Oct 2003 18:45:43 -0700

Since I'm one of the few who advocated this feature 
for the benefit of ULPs (such as iSER), let me add my 
comments.

> is invalidation sufficiently ULP (wrt DDP, e.g., RDMAP)-specific 

Yes, I believe so.  Ultimately, it is the ULP that decides how 
long remote access for an STag (app buffer) should be allowed 
based on application considerations such as an "application transaction" 
(an example "transaction" for iSCSI/iSER is the exchange of 
cmd-data-status for one I/O), and the risk in not invalidating.

>that

Today, the RDMAP spec only provides the semantics and wire protocol
specifics around invalidation.  It is left to the ULPs (of RDMAP) to 
figure out how to use the protocol feature to achieve the desired
results - again they know best when to disallow access to the underlying
buffers.

Now, a natural follow-up question might be why cannot the ULPs 
do the invalidation themselves.  It appeared to us that it's more efficient
to have the auto-invalidation take place in the RDMAP layer, so the
ULP can simply sanity-check the invalidated STag value post-fact (that could
be part of the completion message delivering the Send Message payload), 
rather than having to turn around right away and ask the RDMAP/DDP
layers to do the invalidation.  IOW, one round-trip through the RDMAP/DDP
layers (right after processing an inbound Send Message) is eliminated 
by the auto-invalidation.  In a hardware RNIC implementation where 
the ULP-RDMAP boundary is also the software-hardware boundary,
the savings resulting out of eliminating this round trip are much more substantial.
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668 
Roseville CA 95747
[email-masked]

---

From: steph <[email-masked]>
Date: Sat, 04 Oct 2003 19:38:12 -0400

> the rationale for it being an RDMAP operation rather than a DDP

Caitlin said it: DDP is only about placement and RDMA is about
control.  [But, by the time we're done who knows anymore.]

Invalidation is an important mechanism for CERTAIN applications.  And
given that all RDMA-family protocols over the past 10 or so have
pretty much done it the same way [cough, gasp, not...], I'm sure we
know exactly how to do it.  We should commit it irreversably to gates
and provide no alternatives.

DDP should be a MINIMUM, self-contained placement mechanism.  It is a
base upon which different RDMA-type protocols can be built.  ULPs can
also use DDP, but they should not expect to get a `RDMA-lite'
semantics.  RDMA is whatever fuller placement-based protocol the
community wants for current & anticipated ULPs.  As the layering line
on DDP is pushed down as far as possible, RDMA's service interface
picks up the slack.  For example, DDP should not be saying anything
about how host-side untagged message resources are managed
(i.e. ddp_post_recv()).  But it does.  Oh well.

You can build a variety of invalidation mechanisms on top of DDP.
These could be built by a DDP-targeted ULP, or by an RDMAP (a `general
purpose' DDP ULP).  Invalidation mechanisms defined by an RDMAP become
a candidates for hardware handling (performance optimization) like
write with invalidate.

Steph