Subject: [Tsvwg] RE: Placement "outside of" TCP

From: "Williams, Jim" <[email-masked]>
Date: Wed, 18 Dec 2002 14:45:54 -0500

Joe Touch wrote:
> Caitlin Bestler wrote:
> > > An optimal interface supporting MPA will make validated TCP
> > > segments available for *placement* to MPA out-of-order, but
> > > it does not *deliver* them early. The data is not
> > > "available".
> > 
> > If the data exits TCP, it is delivered.
> > If it is delivered 
> > out of order, 
> > it is a violation of the spec.

This seems like kind of a fine line.  I don't know
how DDP can *place* the data until the DDP
header has been *delivered* to DDP from TCP.
Since DDP must look at, interpret, and act
on this header in order to *place* the data,
this qualifies as *delivery* of the DDP header.

On the other hand, if all this happens within
a block box containing both TCP and DDP,
it is not visible to the external ULP.
(At least it is not visible assuming
a number of caveats yet to be specified.)
Hence, a virtual delivery that doesn't become
real until the ULP observes it.
( Schrodenger's cat likes it. :-)

All this said, though, I don't know why this
matters.  The IETF must decide if this
is an acceptable way to use TCP.   I don't
believe the above arguments are, or should
be, the deciding factor.
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Williams, Jim" <[email-masked]>
Date: Thu, 19 Dec 2002 12:44:46 -0500

> -----Original Message-----
> From: Brian F. G. Bidulock [mailto:[email-masked]]
> 
> 
> Black_David,
> 
> As DDP without TCP framing is not terribly useful, I think we have
> always been talking about the different beast.

We are currently shipping a product that does DDP without TCP
framing.  From the host system perspective, the presense of
absense of TCP framing is not visible, so all advantages that
accrue to the host are the same either way.  (I.e., we think
it's useful.)

> [...] does not alter TCP, and, 
> [...] does alter TCP.

My preference would be to concede that MAYBE it alters TCP
and MAYBE it doesn't, and leave it at that.  If we can
prove beyond a reasonable doubt that it doesn't break 
interoperability, then hopefully that's good enough
and we don't have to resolve the "does or doesn't" argument.
(Just my preference, not sure we can get away with that.)

---

From: [email-masked]
Date: Thu, 19 Dec 2002 13:55:13 -0500

Brian,

> Besides, I'm find it goes against logic to in one breath say that
> DDP/TCP in a black box (with some efficiencies exploited) does not
> alter TCP, and, in the other breath say that  MPA/TCP in a black box
> (with some efficiencies exploited) does alter TCP.

Try this - at the moment:
- DDP/TCP does not alter TCP
- MPA/TCP does alter TCP
- DDP/MPA/TCP does alter TCP
The principle is that MPA's modification of sender behavior
is the alteration to TCP, i.e., "presence of MPA" ==> "alters TCP".

> In fact neither performs cross-layer optimizations (where two already
> useful layers are optimized), but exist solely to attempt to impose
> efficiencies upon TCP.  The argument is that in forcing certain
> efficiencies into TCP, you make it un-TCP.  As it is a different
> protocol internally, even with a different API, use a different
> protocol number.

I believe that's a description of/issue with MPA, not DDP, especially
the "forcing certain efficiencies" part - draft-williams-iwarp-ift-00.txt
describes a way to perform DDP on TCP without "forcing certain
efficiencies".

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: David Borman <[email-masked]>
Date: Thu, 19 Dec 2002 16:49:57 -0600 (CST)

> Date: Thu, 19 Dec 2002 16:07:07 -0600
> From: Caitlin Bestler <[email-masked]>
> Subject: Re: [Tsvwg] RE: [rddp] RE: Placement "outside of" TCP
...
> Any clear "start from scratch" analysis of what DDP required
> from its LLP would come up with something very much like
> SCTP without multi-homing. TCP wouldn't even be close.

Sigh.  Ok, DDP/TCP is just the wrong answer.

> But that misses the whole point. Nobody believes that TCP is
> the ideal transport for RDMA protocols. The goal is to allow
> consenting ULP to experience the benefits of RDMA *without*
> having to change the intermediate network, or deploy a new
> network (such as InfiniBand).

Sigh (again).  Then make DDP/TCP a clean definition and people
will just have to suffer the performance hit when packets get lost
if they can't figure out how to optimize their implementation
in the absence of markers.  That'll give people more motivation
to switch to DDP/SCTP.

If folks really need markers in the TCP data stream, then
MPA/DDP/TCP does not belong as a standard.  At best it should
be an informational RFC (but I might question even that).

> Engineering to be compatible with equipment that is already
> deployed seldom results in pretty solutions -- but the
> effort is important. The investment millions of users have
> placed in their existing network equipment is something
> that should take precedence over our desire to design
> an "elegant" protocol.

Not if that protocol is going onto the standards track.  There
are lots of protocols out there that run over TCP that are not
standards.  It is sounding more and more like this should be one
of them.

So I'm coming to the conclusion that the implementation/engineering
constraints that are being required for MPA/DDP/TCP are so limiting
on the protocol design that the only possible solution for out-of-order
deliver is markers.  And for me, that is sufficient to say that it no
longer passes the sniff test for being on the standards track.

It doesn't matter whether or not the AD/IESG says if you are allowed
to suggest modifications to TCP, your own self-imposed limitations are
more severe, making the issue moot.

But maybe not everyone who supports DDP/TCP feels bound by the same
engineering/implementation constraints.  I'll fully support efforts
for DDP/TCP from people that don't feel that Markers are the only
solution.

		-David Borman, [email-masked]
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: [email-masked]
Date: Thu, 19 Dec 2002 18:24:57 -0500

> be an informational RFC (but I might question even that).

FWIW, the rddp charter says that the TCP mapping of RDMA/DDP would
be an informational RFC.  OTOH, this is a case in deciding not to
put something on the standards track won't stop a significant
number of folks from implementing it.

Thanks,
--David

---

From: [email-masked]
Date: Thu, 19 Dec 2002 18:49:26 -0500

Somesh,

> > > Can you give an example of a mechanism which would satisfy both out
> > > of order placement (direct with no subsequent copy) yet in order
> > > delivery?
> >
> > An async I/O interface is one possible analogue to how an application
> > would use DDP - queue buffers in multiple queues for inbound data,
> > wait for completion events to indicate that I/O has occurred,
> > don't try to look at the buffers in the interim as there are
> > no guarantees as to what will get written to which portions
> > of them when.  Data can be placed into buffers when it's
> > convenient to do so, but in order delivery is enforced by
> > not sending the completion event until it is appropriate to do
> > so.
> 
>   If the out-of-order placement but not delivery principle is to be
followed
>   to the letter, TCP should not look at the DDP header to determine
>   where the data should be placed. What should happen is that
>   the TCP layer places data in shared buffers, and at the time
>   of delivery, provides a linked list of vectors that contain
>   the data.

The only way for the application to determine this is to violate
the interface rule that says "don't look at the buffer until the I/O
is reported as complete", which is part of the async I/O interface
(heaven only knows what sort of data will be in the buffer at any
point in time).  An application that follows that rule can't tell
whether TCP looked at the DDP header because the behavior it can
observe is not distinguishable from an implementation that buffers
the data in TCP and copies to the application buffer just before
sending the I/O completion.  This is headed back into the "black box"
discussion - please see my earlier message on that subject.

Thanks,
--David

---

From: "Williams, Jim" <[email-masked]>
Date: Fri, 20 Dec 2002 17:43:37 -0500

> Black_David wrote:
> 
> Somesh wrote:
> >   If the out-of-order placement but not
> >   delivery principle is to be  followed
> >   where the data should be placed. [...]
> 
> The only way for the application to determine
> this is to violate the interface rule that says
> "don't look at the buffer until the I/O
> is reported as complete".

I am nervous as to where this line of reasoning leads.

The rule says "don't look at the buffer until the I/O is
reported complete.", but WHICH buffer should the application
NOT look at?

The simple answer is "Don't look at the buffer specified
in the undelivered DDP header."  But the application is not
supposed to know what's in that undelivered header.

Another answer is "Don't look in ANY buffer to which the
local RNIC has write access."  The sounds safe, but is 
overly restrictive.  It requires the application to 
explicitly revoke the RNIC's write access to any buffer
before looking at it.  I wouldn't rule this out of hand,
but the restrictive nature would be a concern.

A third answer is "Don't look in any buffer that I 
(the local application) and the remote
application have agreed that the remote 
application is allowed to write to."  This looks promising,
but it's a bit vague and abdicates responsibility
to each application to get the right on its own.
It also involves trusting the transport not only to 
deliver correct data, but to insure the correctness
of undelivered data (the DDP header) as well.

Perhaps the third answer is the right one, but
we need to get some agreement on that and start
nailing things down.