Subject: RE: [rddp] Alignment and TCP modification

From: "Williams, Jim" <[email-masked]>
Date: Mon, 6 Jan 2003 14:40:27 -0500

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> 
> So, I believe the primary issue before the WG is whether to accept:
> 	- The RDDP protocols must allow implementations over
> 		unmodified TCP/IP stacks.
> as a requirement.

I don't believe anyone has proposed or recommended otherwise.
(MPA states an MPA-aware sender SHOULD align, but even that
is not contradictory to the above.)
I thought all alignment proposals were optional on the 
part of the sender.  Alignment is potentially exploitable by
the receiver if it exists, and some mechanism should
allow the receiver to determine if alignment is present.
Maybe I am missing the point here.

> 	- This would imply that a sender using an unmodified stack
> 		could send RDDP payloads that are directly placeable
> 		by an RDDP receiver.

By directly placeable, you mean placeable even if received out
or order?  If so, I don't understand how an unaligned RDDP PDU is
directly placeable, and as you stated, unmodified TCP stacks
can't do alignment.

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Tue, 7 Jan 2003 09:25:06 -0500

Jim,

> Maybe I am missing the point here.

The point is that "optional" (MAY) is a considerably weaker
requirement than SHOULD (see RFC 2119).  I believe that an
unmodified TCP stack cannot comply with a "SHOULD align"
requirement solely on the basis that the an implementer does
not want to or cannot modify the stack, in contrast to a
"MAY align" requirement.

In turn, a "MAY align" requirement for senders would make it
incumbent upon receivers to do a "reasonable job" for
unaligned traffic (and there are a range of views in the WG
on what is "reasonable"), in contrast to a "SHOULD align" that
would allow receivers to take the attitude that "the sender
ignored a SHOULD, and hence deserves the lousy performance that
it's going to get".

This is not just a wordsmithing issue, as it is strongly related

> directly placeable, and as you stated, unmodified TCP stacks

I believe so.  If we accept the requirement, I am concerned about
problem magnification - essentially, transient network congestion
causes TCP to start sending unaligned, which in turn causes serious
performance problems on servers that relied on aligned transmission
for performance, thus turning a network problem that TCP can readily
ride out into a serious overall system performance glitch.  While
implementers cannot be prevented from doing this, one can use
carefully drafted and well explained SHOULDs and SHOULD NOTs to
warn of the perils lurking here.

Markers and similar mechanisms are among the ways to obtain direct
placeability of unaligned RDDP PDUs, although I don't see the need
for every RDDP PDU to be directly placeable under all circumstances
(cf. previous list discussion on why there's no such thing as a truly
bufferless [R]NIC).

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: [email-masked]
Date: Tue, 7 Jan 2003 17:59:54 -0500

Jim,

> > > [...]
> If a sender MAY be MPA-aware, and an MPA-aware sender
> SHOULD align, then we have an overall MAY.  This was how
> I interpreted the draft.  However, your
> point is well taken, and the draft could be a bit clearer on this.

That's one side of it, the other side (ellipsed in your message)
is whether the receiver is under some obligation do a "reasonable job"
of RDDP on traffic that arrives unaligned and what that means.  If
the only way to avoid "MPA-aware sender SHOULD align" is to not be
MPA-aware, and the sender not being MPA-aware disables RDDP at the
receiver, then RDDP does not run over unmodified TCP stacks (oops).
In defense of the MPA draft, I believe it intended to separate the
MPA additions to the bytestream from MPA-aware TCP, but the MPA
with unmodified TCP case doesn't get sufficient attention.

> > placeability of unaligned RDDP PDUs [...].
> 
> The data can't be placed until the header is received.  In the
> unaligned case, at best only part of the data can be placed, even
> with markers or similar mechanism.
> 
> Doing partial packet placement, while theoretically possible,
> has some serious practical concerns.
>
>   1. To be of any value, it implies the ability to free partial
>      buffers for re-use, while keeping the unplaced data.
>      Or else copying the unplaced fragment elsewhere to allow
>      freeing the buffer.  Either causes serious implementation
>      complexities that are avoided in the aligned case.

I think that's overstated.  There are "complexities", but they
don't entail either copying or partial freeing.  For example,
one could use reference counting to not free a buffer until all
of its pieces have been placed.
 
>   2. Since each supported connection may have an associated
>      unplaced fragment, the total data buffering required
>      now scales linearly with the number of active connections.
>      This is fundamentally different than the aligned case.
>      Worse, if there are multiple gaps, there may be 
>      multiple unplaced fragments on a single connection.

Yes, but that's still better than connection count x bandwidth x
round trip time for the case in which a TCP retransmit has to be
waited for before placement can occur.

> I do not believe that direct placement of unaligned PDUs 
> achieves the objectives that the proponents of direct
> placement are looking for.

No comment, for further discussion.

Thanks,
--David

---

From: [email-masked]
Date: Tue, 7 Jan 2003 18:13:02 -0500

Steph,

> I'm too woozy right this second to completely understand the details
> of your consensus request, so I'll just charge ahead.

By all means.  This consensus request is a bit tricky as I'm trying
to establish consensus on a requirement, some of whose implications
are not immediately or plainly obvious.
 
> Perhaps stating the obvious, it seems ridiculous to require RDDP run
> on a TCP that performs sender alignment.  RDDP must be able to run on
> an unmodified TCP.

I'll note you as being in favor of the requirement.

>  It seems equally ridiculous to expect the receiver
> to hypothesize but verify sender alignment without any information
> about whether the sender is even trying to provide it.  An unmodified
> sender is not likely to end up aligning by accident.

I'm not sure about that.  For a request/response ULP where there's
not enough traffic to keep data flowing in both directions all the
time, alignment by accident may well happen based on draining the
send queue at various points (esp. if the Nagle optimization is
disabled which seems to be getting an increasing level of recommendation
these days).  iSCSI, NFS, and CIFS come to mind as examples of such
protocols for some reason [;-)], but HTTP can also behave this way.
 
> That leaves the endpoints negotiating whether the sender will try, and
> if so, the receiver will verify, correct?

That's certainly one way to approach the problem.  My impression of the
MPA draft is that it went in the direction of always trying to optimize
for alignment in the hope that modified senders will help out by aligning
in situations where unmodified ones may fail to align, but I'll be happy
to have the MPA authors clarify this.  Some sort of negotiation-like
mechanism could also help with the "avoidance of framed vs. non-framed
traffic confusion" issue that the WG has to resolve.

Thanks,
--David

---

From: "Williams, Jim" <[email-masked]>
Date: Fri, 10 Jan 2003 16:17:07 -0500

Caitlin Bestler wrote:

> BTW, am I correct that the only spec issue that depends upon
> these questions is that of the header CRC?

CRC question is secondary.  Primary goal was to move 
incrementally closer to consensus on alignment and markers.
Answers to these questions may or may not help, but
that was the goal.
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Fri, 10 Jan 2003 14:56:08 -0800

The answer to the second question is a *loud* YES. Hence, the 
answer to the first question is #3 and possibly #3+, if there 
were one.

Some related comments:

* We seem to keep coming back to the *assumption* that RDMA is 
  only for data center or machine room scenarios. The 
  applicability of RDMA/RDDP is *far wider* than that. In the
  life time of this IETF standard, we are talking about 
  convergent NICs that will make sense for any application 
  that needs movement of ?significant? amounts of data. It may
  make sense to agree upon a list of compelling applications 
  for the next 20+ years.

* Ironically, it is the data center application that, 
  *by-design*, has the luxury of negligible out-of-order-ness
  and/or packet drops. Since I may be delving in a few 
  not-so-obvious and possibly debatable assumptions, I will 
  take this topic up in a seperate thread.

-Shridhar Mukund

> From: Williams, Jim [mailto:[email-masked]]
> Sent: Friday, January 10, 2003 8:44 AM
> To: [email-masked]'; [email-masked]
> Subject: RE: [rddp] Alignment and TCP modification
> 
> 
> I would say yes to this.  A few more thoughts follow.
> 
> What is the required performance of an RDMA receiver when
> receiving data from a transmitter using an unmodified TCP/IP stack?
> 
>     1.  Only functional correctness matters, performance is 
> not an issue.
> 
>     2.  Performance must be as good as conventional (non-RDMA) TCP.
> 
>     3.  Performance must be significantly better than conventional
>         (non-RDMA) TCP otherwise RDMA should not be used in this case.
> 
> A second question is:
> 
>        - Is the case of a large number of slow clients using
>          unmodified TCP/IP stacks talking to a fast server
>          using an RDMA accelerated NIC an important design point?
> 
> If the answer to the second question is YES, then I believe
> that the answer to the first question is #3.
> 
> If the answer to the second question is NO, then I believe
> the answer to the first question is #1, since in practice
> no one will use RDMA over an unmodified TCP/IP stack.

---

From: "Williams, Jim" <[email-masked]>
Date: Mon, 13 Jan 2003 15:45:59 -0500

If software implementations are common and important,
that suggests there is value in making the CRC optional.
For hardware implementations, once the CRC is implemented,
it might as well be used always.

However for software there is significant overhead for the
CRC, and some reasons why it might be unnecessary.

  1.  IPSec authentication is in use.

  2.  Connection is on a private network with good quality
      switches/routers and good link CRCs.  So TCP checksum
      is enough.

  3.  ULP does its own end to end integrity check.

> From: Mukund, Shridhar [mailto:[email-masked]]
> Sent: Friday, January 10, 2003 5:56 PM
> To: Williams, Jim; [email-masked]'; [email-masked]

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Mon, 13 Jan 2003 14:15:11 -0800

and,

4. Several applications can tolerate TCP chksm escape.
   ( I am assuming that you are talking only about payload
   CRC and not header CRC )

> Sent: Monday, January 13, 2003 12:46 PM
> To: [email-masked]

---

From: "Uri Elzur" <[email-masked]>
Date: Tue, 14 Jan 2003 10:38:26 -0800

I don't think we have consensus over having a split CRC. On the contrary, one CRC has many advantages (no intention to re-has here). It will become more evident after the direction for alignment is set. If alignment is assumed, no need for two CRCs.

Uri

-----Original Message-----
From: Caitlin Bestler [mailto:[email-masked]]
Sent: Monday, January 13, 2003 2:46 PM
To: Williams, Jim
Cc: 'Mukund, Shridhar'; [email-masked]
Subject: RE: [rddp] Alignment and TCP modification

On 1/13/03, Williams, Jim wrote:

>> 
>> > However for software there is significant overhead for
>> > the CRC, and some reasons why it might be unnecessary.
>> > 
>> > 2.  Connection is on a private network with good
>> > quality switches/routers and good link CRCs.  So TCP
>> > checksum is enough.
>> > 
>> 4. Several applications can tolerate TCP chksm escape. (
>> I am assuming that you are talking only about payload
>> CRC and not header CRC )
>
>I was suggesting both header and payload CRCs be optional,
>however should be separate option, as you point out some
>may need header CRC but not payload CRC. 

Deciding to rely upon ULP validation of the payload is
valid. But if the DDP Header is received in error, the
payload may have been placed anywhere that is write-enabled
for the receiving ULP.

Allowing omission of a CRC covering the DDP Header strikes
me as very dangerous, and possibly a security hole.

If there is a consensus to support splitting CRC coverage
between header and payload, I could see making payload
coverage optional. But the extra complexity is just one more
argument for sticking with a single CRC.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: "Mukund, Shridhar" <[email-masked]>
Date: Tue, 14 Jan 2003 15:23:03 -0800

Dear colleagues,

In the next several email sub-threads, I intend to make the
following main arguments with hopefully ample support arguments 
- so help me god :-)

A. Alignment breaks TCP flow control and *semantics* in very 
   *fundamental* ways. The SAR portion of DDP is an *artifact*
   of the current MPA solution rather than being an *intrinsic* 
   complexity of the defacto problem statement, say "the quest".
   THE QUEST: The desire to direct-place every byte of every 
   packet even if it arrives out of order (and unaligned).

B. Contrary to intuition, SAR+MPA complicates design. In fact,
   it increases the buffering requirement instead of reducing it! 
   I will stay implementation neutral, when discussing this.

C. Lack of alignment is and will be sufficiently common that it
   should not be treated as an error case. Fluid flow (byte stream)
   property is probably the most important *intrinsic nature* and 
   strength of TCP.

D. For the fear of being stoned to death:-), I am going to assume
   "the quest" to be sacrosanct. It is illustrative to evaluate the
   incremental value proposition compared to say 
   iSCSI-like-marker-and-DDP (the baseline alternative). On the 
   other hand, it is also not necessary to break TCP's back bone 
   for efficient and high-performance realization of "the quest". 
   
   The suggestion is to perturb only the sheet-metal part of ROI.
   THE GOAL: To take RDDP TCP mapping from a state of non-solution,
   with sufficient risk of infant mortality, to TCP-like longevity
   and ubiquitous ness.   
   Call for action (pl. save it for later):
	D1. Move the SAR part from DDP spec to SCTP adaptation.
	D2. Merge IFT+MPA and upgrade draft not to require 
          alignment. ?How?
   I have no intention to derail the monumental efforts that
   have gone into RDMAP, Verbs, and DDP. I am humbled by that 
   effort, esp. since I personally did not make any contribution 
   to it.

-Shridhar Mukund