Subject: [rddp] Alignment and TCP modification

From: [email-masked]
Date: Thu, 2 Jan 2003 15:21:58 -0500

In response to my request for explanations of opposition to
requiring sender alignment:

	So, let me ask those three people, plus anyone else
	who is opposed to a general approach of requiring
	sender alignment to send me a brief email off the list
	summarizing their viewpoint and the reasoning behind it.

I received a total of six responses, five of which expressed
varying degrees of opposition to a sender alignment requirement:
	- Somesh Gupta, Caitlin Bestler, David Borman, Randy
		Stewart and Shridhar Mukund
and one that indicated that alignment could be an acceptable
compromise if it could be shown to be of practical value:
	- Jim Williams
I'll leave it to these people to explain their views in further
detail if they think it's warranted (in some cases, what was sent
to me was a summary of what had already been posted to the list).

I'm struck by the fact that a single theme was in common among
the five individuals who expressed opposition, namely:
	- The RDDP specifications need to allow implementations
		over unmodified TCP/IP stacks.

The primary rationales for this are:
(1) An unmodified stack will send unaligned in some circumstances
	that cannot be avoided unless one resorts to truly draconian
	and unacceptable measures such as a ULP-level stop-and-wait
	protocol (ULP ACK between sends drains TCP send queue ensuring
	that next ULP send is aligned by TCP).
(2) Even for a hardware-accelerated ULP server implementation,
	software ULP clients will be common and important.  Implementing
	the RDDP functionality in software over an unmodified stack for
	ULP clients in this scenario is important to reduce barriers
	to adoption/use.  For example, this enables all of RDDP to be
	implemented in user mode over an existing stack.
(3) The ability to run over an unmodified stack makes the RDDP
	protocols far more applicable and hence amenable to
	widespread adoption, including the above ULP client scenario.

Beyond this, I saw a divergence of opinion on how much attention
implementers should pay to aligned vs. unaligned traffic beyond
agreement that it should not be treated as an error case.  This
divergence ranged from "most traffic will be aligned, but unaligned
traffic should not run any slower than ordinary TCP traffic" to
"unaligned traffic will be common enough that implementations
need to optimize for it, even if most/all senders send aligned"
(quoted text is my paraphrase in both cases).  Fortunately, the WG
does not need to settle this issue - implementers are always free
to optimize whatever cases they believe are important.  Some of
the emails also raised other issues that I'll leave it to those
involved to decide whether to discuss further on this list.

So, I believe the primary issue before the WG is whether to accept:
	- The RDDP protocols must allow implementations over
		unmodified TCP/IP stacks.
as a requirement.  If this should be accepted as a requirement,
then sender alignment cannot be required of all senders (this
wouldn't rule out negotiating it in some fashion).  Here are a
few more implications:
	- Unmodified means 100% unmodified, not even "minor" changes.
	- This would imply that a sender using an unmodified stack
		could send RDDP payloads that are directly placeable
		by an RDDP receiver.
	- There don't appear to be any receiver implications, as
		although an optimized RDDP implementation will likely
		use a modified stack, I haven't seen any proposal that
		requires such modifications (e.g., disallows the brute
		force technique of having TCP buffer as usual and then
		running RDDP functionality on the in-order TCP bytestream).
	- The practical meaning of "allow" would be that an
		implementation is possible without violating any MUST,
		MUST NOT, SHOULD, or SHOULD NOT requirement.
The latter bullet is based on my view that "I don't want to modify
an existing stack" is not a sufficient reason by itself to disregard
a SHOULD or SHOULD NOT requirement.

All of this is open to discussion (including the immediately
preceding SHOULD/SHOULD NOT implication).  That should be sufficient
to get further discussion going ;-).  I would ask that those
opposed to accepting the requirement start from the above numbered
rationale and explain what they believe to be wrong/unconvincing/
missing/etc.

Thanks,
--David

----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953 **NEW**     FAX: +1 (508) 293-7786
[email-masked]        Mobile: +1 (978) 394-7754
----------------------------------------------------
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Sat, 4 Jan 2003 12:35:08 -0600

On 1/2/03, [email-masked] wrote:

>
>Beyond this, I saw a divergence of opinion on how much
>attention implementers should pay to aligned vs. unaligned
>traffic beyond agreement that it should not be treated as
>an error case.  This divergence ranged from "most traffic
>will be aligned, but unaligned traffic should not run any
>slower than ordinary TCP traffic" to "unaligned traffic
>will be common enough that implementations need to
>optimize for it, even if most/all senders send aligned"
>(quoted text is my paraphrase in both cases). 
>Fortunately, the WG does not need to settle this issue -
>implementers are always free to optimize whatever cases
>they believe are important.  Some of the emails also
>raised other issues that I'll leave it to those involved
>to decide whether to discuss further on this list.
>

I'm firmly in the camp that favors ensuring that unaligned
operation is *possible* but expecting that it will be the
exception.

It is important to keep in mind that RDDP provides two basic
benefits: direct placement of incoming data and eliminating
false ordering requirements.

Features such as alignment support implementation of direct
placement in hardware. These are important benefits, but
they are not necessary to justify the deployment of RDDP.
A ULP can still benefit from the relaxed ordering rules
and/or from merely having one end of a session with hardware
accelerated data placement.

The prime scenario for this to me is an RDMA enabled file
server dealing with a large number of clients. The clients
may not have enough RDMA specific traffic to justify
hardware assist. But the server can still benefit by being
able to use RDMA at its end, because it does have enough
traffic to justify hardware placement.

The question then becomes whether protocol enhancements
should be done to support optimal hardware implementation
when faced with high volume out-of-order delivery of
non-alligned traffic.

My estimate remains that the overwhelming majority of
connections, even long-haul connections, will not break
alignment. Therefore lower-cost hardware that has a limited
capacity to optimize non-aligned packets will still be of
considerable value.

With those assumptions, I think most people would agree that
addign complexity and overhead to the protocol (such as a
second CRC for just the header) is not justified. Does
anyone agree that the second CRC provides no benefit when
alignment is maintained?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Stephen Bailey <[email-masked]>
Date: Tue, 07 Jan 2003 16:28:36 -0500

David,

I'm too woozy right this second to completely understand the details
of your consensus request, so I'll just charge ahead.

Perhaps stating the obvious, it seems ridiculous to require RDDP run
on a TCP that performs sender alignment.  RDDP must be able to run on
an unmodified TCP.  It seems equally ridiculous to expect the receiver
to hypothesize but verify sender alignment without any information
about whether the sender is even trying to provide it.  An unmodified
sender is not likely to end up aligning by accident.

That leaves the endpoints negotiating whether the sender will try, and
if so, the receiver will verify, correct?

Steph

---

From: Caitlin Bestler <[email-masked]>
Date: Wed, 8 Jan 2003 11:20:14 -0600

On 1/7/03, Stephen Bailey wrote:

>
>Perhaps stating the obvious, it seems ridiculous to
>require RDDP run on a TCP that performs sender alignment. 
>RDDP must be able to run on an unmodified TCP.  It seems
>equally ridiculous to expect the receiver to hypothesize
>but verify sender alignment without any information about
>whether the sender is even trying to provide it.  An
>unmodified sender is not likely to end up aligning by
>accident.
>
A DDP/MPA stack implemented on top of a conventional sockets
interface could be attempting to achieve alignment by
flushing. This could succeed most of the time, but fail when
the unaltered stack had reasons to ignore the user level
hint (such as retransmissions or congestion).

Similarly a TCP-spoofing middlebox (that terminates and
re-orginates the TCP connection) might be deployed even
without a severe MTU change. For example, the pacing
characteristics of the two networks are radically different.
Such a middlebox might pass *most* TCP segments through
unmodified, but reserve the right to consolidate them on
retransmits, etc.

Administrative methods could be deployed to allow a receiver
to *know* certain connections will be aligned. For example a
list of "safe" netmasks could be applied. But I do not think
there is anyway to *dynamically* determine that a given
connection is "safe" and is guaranteed to maintain alignment
from that point forward.

Caitlin Bestler