Subject: [rddp] STag Invalidation and RDMAP vs DDP question

From: Caitlin Bestler <[email-masked]>
Date: Mon, 27 Jan 2003 18:17:50 -0600

What are the requirements for the timing of STag
invalidation by the RDMAP layer?

There are two scenarios that could make this something
requiring careful specification.

a) The immediately following message attempts to use
   the same STag. If the DDP layer MUST reject it, then
   the DDP layer will have to wait synchronously for the
   RDMAP layer to complete processing of the prior
   invalidating send message. If you are actually
   considering making these real layers this would
   be a problem.
   
b) The prior message was an RDMA Read request. Will the
   read have to remain valid until it completes, because
   it was queued before the invalidate?
   
I believe a clean layering is possible without disrupting
ULP expectations that badly, if the following caveats are
added.

1) The RDMAP engine must ensure that it has extracted all
information required from an STag to complete any pending
RDMA Read before it processes the STag Invalidation.

2) The RDMAP engine must not report completion of an
invalidating send to the ULP before the STag has in fact
been invalidated.

3) The RDMAP engine will declare an error when it detects an
RDMA Write to an invalidated STag, even if the DDP layer had
allowed the placement to occur.

The net effect is that the memory is exposed for a very
brief additional period, but that the exposure must be
eliminated before the consumer is told of the invalidation.
If something did slip into the crack, the consumer is
told about it as well.

Without this caveat, I believe that the DDP and RDMAP
"layers" would have to be so synchronous with each other
that they would not in fact be layers.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Fri, 31 Jan 2003 18:10:04 -0700

Caitlin,

The only requirement is that the RDMAP for STag invalidation is that STag is to be invalided before RDMAP delivers the Send with Invalidate to the ULP.

The purpose of the Send with Invalidate is to invalidate the STag so that the ULP can reuse the STag either on the same connection or another connection. That protection is accomplished by ensuring that the STag will be invalidated before the completion of the Send with Invalidate Message is delivered to the ULP.

For scenario a, a Send with Invalidate Message doesn't protect against a following message in the data stream changing the data because DDP is allowed to do out of order placement. If the remote peer was going to corrupt the data it was writing to the tagged buffer, it could write the wrong data in the first place or it could delay sending the Send with Invalidate. There is therefore no reason to protect against an RDMA Write to the tagged buffer in the stream right after the Send with Invalidate as long as the data is placed before the Send with Invalidate is delivered to the ULP.

If scenario b occurs, something is broken. A ULP should not issue a Send with invalidate for an STag when there are uncompleted RDMA Read Requests. There is no ordering between completion of RDMA Read Requests and completion of Sends. If scenario b occurs the STag will be invalidated and DDP may get an Invalid STag error if it hasn't finished processing the RDMA Read Response. 

Regards,
Pat

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Monday, January 27, 2003 4:18 PM
To: [email-masked]
Subject: [rddp] STag Invalidation and RDMAP vs DDP question

   

Caitlin Bestler