Subject: [Tsvwg] "DO IT RIGHT", or "DON'T DO IT AT ALL"

From: "Williams, Jim" <[email-masked]>
Date: Thu, 5 Dec 2002 14:05:23 -0500

The subject under discussion here is transforming
TCP into a record aligned transport to allow receivers
to do some level of ULP processing prior reassembly
of the TCP stream.  The specific goal is to enable
an RDDP protocol to directly place incoming
data even if TCP segments arrive out of order.

At least two drafts have been submitted which
propose mechanisms:

     draft-ietf-tsvwg-tcp-ulp-frame-01.txt [tuf] (expired)
and
     draft-culley-iwarp-mpa-00.txt [mpa]

In Atlanta I argued the case for "DON'T DO IT AT ALL".
The case being that TCP is a stream protocol and should
not be used otherwise.  SCTP is a record protocol and
should be used when a record protocol is needed.
And that "fixing" TCP rather than switching to SCTP
is the wrong architectural direction.

However in that aftermath of Atlanta I fear that
my argument is losing within the RDDP group, so I would
like to at least explore the alternative.

Under the heading of "DO IT RIGHT", both drafts
fall a bit short for the same reason.  The reason being
they both went to great lengths to minimize any changes
to TCP.  So I would like to explore the possibility that
changing TCP "just a little bit" may not be the
right approach to solving the problem, and that TCP
should be changed to the extent necessary to solve the
problem the right way.  Of course, this needs
to be done in a way that does not break interoperability.

The guiding architectural principal for "doing it
right" is that the receiver should have some level of
guarantee that it will always get record aligned TCP
segments, and that any unaligned segments can and should
be considered an error case, resulting in a closed
connection.

Both [tuf] and [mpa] state
that the receiver might or might not get record aligned
segments, and they both add another layer of protocol
processing to allow the receiver to determine whether
the alignment attempt was or was not successful.  They
both require that the receiver to support the dual 
complexity of handling both aligned and unaligned
segments on a given connection.

To me, this seems to have the type of undesirable 
complexity often associated with ill advised, ad hoc
solutions.  I would encourage that some thought
be given to a more cleanly architected solutions
even if the down side is greater impact on TCP
bahavior.

There are two fundamental problems with guaranteeing
record alignment to the receiver.

  1.  How can the sender always align.

  2.  How can middle-boxes be prevented from
      re-segmenting the TCP stream.

With respect to 1., both drafts state that alignment
will usually be done, but list a number of exception
cases under which the sender will not attempt to 
align.  

I would propose as part of "doing it right" that
a sender which agrees to align must ALWAYS align
records with TCP segments.  I will not attempt
now to list here all the implication of this, but 
the summary is that this will require
small but substantive changes to TCP
sender behavior, and that these changes, if done
right, will not break interoperability with 
existing implementations.  (However this is
a tricky area, and until the work is done, 
it is not possible to be sure.)

With respect to 2., I would propose that senders
that agree to align should announce this intention
by adding a newly defined RECORD_ALIGNED TCP option
to the SYN segments used to set up the connection.

This is a reasonable architectural solution in
that middle-boxes that want or need to re-segment
the TCP stream should not include this RECORD_ALIGNED
option in their outgoing SYN segments.

It is admittedly only a partial solution in a 
practical sense.  Undoubtedly, some fraction
(hopefully small) of existing re-segmenting
middle-boxes will transparently pass through
unrecognized TCP options.

However, in practice this will not be a big enough
problem that it should drive protocol design in a
major way.  Middle boxes that re-segment, and also
send a TCP option agreeing not to re-segment
are engaging in aberrant and erroneous behavior,
and will cause connections to quickly close in error.
The result is either they will be fixed, or
the affected receivers will be configured to
not assume alignment (ignore incoming RECORD_ALIGNED
options).

It is hoped that CRCs used for data integrity
will be sufficient to also protect against aberrant
middle-box behavior.  One can certainly construct
hypothetical scenarios that defeat this, but
these are unlikely to be a problem in practice.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Thu, 5 Dec 2002 12:28:10 -0700

Jim,

Please go back to "DON'T DO IT AT ALL".  SCTP is the superior approach
here, resplendent with HOL blockage avoidance with streams and
multi-homing interface/network redundancy.  With very minor extension to
SCTP, DDP can be directly supported by SCTP adding performance gains in
the face of noise and failure that cannot be approached with TCP's
ordered pipe view of the world.

--brian

-- 
Brian F. G. Bidulock    ¦ The reasonable man adapts himself to the ¦
[email-masked]    ¦ world; the unreasonable one persists in  ¦
http://www.openss7.org/ ¦ trying  to adapt the  world  to himself. ¦
                        ¦ Therefore  all  progress  depends on the ¦
                        ¦ unreasonable man. -- George Bernard Shaw ¦
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 Dec 2002 14:38:35 -0800

> like to at least explore the alternative.

I'd say "DON'T BOTHER AT ALL". Here's why:

	1- anything that really does something useful
	is going to require EITHER a change to
	TCP semantics (the SEND/RECEIVE etc. API) OR
	to default to current semantics

	2- changing semantics of the default case is
	simply not an option. There's too many
	critical (e.g., financial, commerce, etc.)
	systems that are based on existing semantics.
	We don't know what will break or when, and
	setting up a wide-scale gamble simply isn't
	appropriate.

	3- changing semantics of a non-default case
	begs the question: why not just compile in
	a different protocol? SCTP, DCP, or write
	your own on UDP.
	
	The answer I've been hearing is "we need
	it now, and need it to work for legacy apps".
	But that means you also need legacy semantics,
	which means #2 applies.

Finally, I'd appreciate hearing a discussion of why this should all work 
that doesn't include some glaring errors. E.g., at the last meeting we 
all heard about how TCP with Nagle off (NODELAY) sends packets that 
match what the send calls issue. That simply isn't the case, never was, 
and cannot be assumed regardless. E.g., how do you know the path MTU 
didn't change between when you checked it/issued the write and when the 
packet is emitted?

Many of the other debates about these other protocols ignore the fact 
that the TCP spec is defined in terms of two interfaces - one below (the 
packets) and one above (the API, in terms of generic 
SEND/RECV/OPEN/CLOSE etc.) The API appears to be ignored in most cases.

...
> even if the down side is greater impact on TCP

How about realizing that: these are TWO different transport protocols, 
and deserve two different protocol numbers?

> to the SYN segments used to set up the connection.

Good luck on that. Middleboxes exist because the middlebox owners want 
them to, and often don't want to be seen. They already violate the E2E 
argument, and many aspects of the core specifications of the Internet 
(1122,1123, etc.).

Now you expect them to respect a flag you put in the header. Why do you 
expect them to respect a new spec, when the don't respect the existing ones?

...
> and will cause connections to quickly close in error.

Yeah, right. Or the will cause the applications to break, and not work 
at all, as NATs often do. Then what?

> not assume alignment (ignore incoming RECORD_ALIGNED

How? When you're at a meeting and need to get to your email, then what? 
Did the NAT boxes that hit you last time magically disappear, or did 
your end or your servers' get reconfigured to work?

This is not a useful path to spend further energy, IMO.

Joe

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 5 Dec 2002 18:32:10 -0600

The goal for DDP over TCP has always been twofold:

- to allow DDP to work the way it was intended to work.

- to allow DDP to work with the IP networks as they are,
  without any adverse effect on any middleboxes, or
  any dependency on network architecture.
  
I would maintain that however messy MPA might be, it meets
both of these objectives.

SCTP is indeed a superior protocol to MPA/TCP for DDP, as it
is for many applications.

Realistically, however, if DDP relied solely upon SCTP it
would *not* meet the objective to work with IP networks *as
they are*. There are too many environments where middleboxes
have no SCTP capability, or at least no SCTP ability that
has been enabled.

Structuring DDP so that it does not work with out-of-order
delivery does not work either. Dropped packets are not fluke
occurences. They are predictable in virtually any network.

A high bandwidth performance cannot be listed as
"sustainable" if a single drop destroys it. Therefore to
meet the targeted rates as sustainable rates, a DDP RNIC
would have to have large buffering capacity to hold all
of the post-drop segments.

There are at least two modifications to MPA that might make
sense: packing of ULPDUs and dual CRCs. I favor the first
one, but not the second. I do not believe the other
'options' under discussion are in fact viable solutions.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 Dec 2002 17:00:17 -0800

> have no SCTP capability, or at least no SCTP ability that

Good. But then DDP doesn't work with networks as they are either. It has 
to be installed on both ends.

I.e., as I've said before, this is a new transport protocol. Call it 
that. Pick another number. 6 is taken.

Joe

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 5 Dec 2002 19:51:24 -0600

On 12/5/02, Joe Touch wrote:

>> I would maintain that however messy MPA might be, it
>> meets both of these objectives.
>> 
>> SCTP is indeed a superior protocol to MPA/TCP for DDP,
>> as it is for many applications.
>> 
>> Realistically, however, if DDP relied solely upon SCTP
>> it would *not* meet the objective to work with IP
>> networks *as they are*. There are too many environments
>> where middleboxes have no SCTP capability, or at least
>> no SCTP ability that has been enabled.
>
>Good. But then DDP doesn't work with networks as they are
>either. It has to be installed on both ends.
>

That is hardly relevant, as that the application has to be
installed at both ends first.

Most importantly, DDP does not have to be installed by the
system administrator. It *can* run cleanly on top of TCP or
SCTP in user mode.

Obviously it will only hit full speed when kernel level
drivers and/or enhanced hardware is installed. But the
ability to deploy clients at the application layer is
one of the major advantages of using an *existing*
transport. Which is why that goal was included in the
charter.

>I.e., as I've said before, this is a new transport
>protocol. Call it that. Pick another number. 6 is taken.
>

Any new transport protocol would be years behind SCTP in
terms of middle-box support. Relying on SCTP would make far
more sense.

Defining additional IP transports will not cause them to be
deployed. Indeed it is likely to get some middlebox vendors
to simply give up in frustration and retreat to only
supporting TCP and UDP.

More fundamentally, DDP does *not* define many things that a
transport protocol does. By riding on top of *existing*
transport protocols it not only speeds deployment of DDP, it
ensures that improvements in the underlying transports will
continue to service the widest range of applications.

By riding on existing transports, DDP avoids duplication of
efforts and solutions related to:

- Connection Establishment: including related load balancing
  and admission control.
- Quality of Service controls.
- Congestion control.
- Firewalling and security.

- 

Caitlin Bestler

---

From: Joe Touch <[email-masked]>
Date: Thu, 05 Dec 2002 20:44:48 -0800

> installed at both ends first.

My point is that if you're going to modify the ends, install SCTP.

> system administrator. It *can* run cleanly on top of TCP or

To what end? That's just a waste. The alignment benefit is useful only 
(or 99.9% of the cases) when installed in the kernel/enhanced hardware. 
At that point, use SCTP.

> transport. Which is why that goal was included in the

Then run a layer that emulates TCP to the top and uses TCP or SCTP 
underneath, as needed. Prove that works first, then prove that you need 
more than that.

> terms of middle-box support. Relying on SCTP would make far

Perhaps.

> to simply give up in frustration and retreat to only

Perhaps as well. All good reasons to do a layer as above, rather than 
creating another protocol that itself is prone to breaking at middle-boxes.

> efforts and solutions related to:

All good ideas. Except for overriding what the transport protocol thinks 
is the order of delivery. That's when it tries to undo something the 
transport layer already does.

Joe

---

From: David Robinson <[email-masked]>
Date: Thu, 05 Dec 2002 15:15:04 -0600

> be considered an error case, resulting in a closed

Under the guise of "doing it right", I have to ponder if
alignment is the requirement or just a possible solution?

In a stream of data that ultimately can be reordered, resegmented,
and realigned the problem is how to find a header so we
can correctly place the data with minimal buffering. It
is clear that if we get the data portion before its header
we must either drop that segment or buffer it because
it cannot be placed correctly. The problem in an
unmodified stream is what to do with all the subsequent
data until the missing segment arrives to provide the
header necessary to reestablish framing.

One solution is to control alignment so there is a
reasonable expectation that there is a reasonable
expectation that the start of a subsequent segment
will be a header and framing can be reestablished.

If we are going to entertain "changing" TCP, how can
we do it with minimal invasiveness and most benefit?

The first assumption is that the actual record marking
or framing information will be contained within the
TCP stream. We already need headers to define the
placement control information so adding new redundant
out of band information is not needed.

The second assumption is that reordering and retransmission,
is relatively rare. It will not likely occur on every
segment but may occur on order of every few 100 segments.
Resegmentation with the "right" middlebox may occur on
every segment, as in standard TCP it should go unnoticed
and have minimal impact.

As an alternative solution, create a TCP option
RECORD_MARK which contains the sequence number
of an inband framing header. In some past proposals
such an option was offered to mark every frame which
has some performance and other implications. If
the sender is not required to add the option for every
frame, and instead uses it say every 100 or so frames, the
impact on the network path can be minimal.  The tradeoff
is that the receiver in the face of a lost or reordered
segment containing a framing header, must buffer until
it gets the segment containing a RECORD_MARK option. It
may be useful for the sender and receiver to initially
negotiate what the available buffer space is to determine
the necessary RECORD_MARK interval to avoid dropped segments.

The advantages of this sort of technique is that for most
segments will not contain a TCP option and thus will not
affect fast processing paths. There is no impact on implementations
purely layered on top of TCP. While it doesn't eliminate the
need for buffering, the amount of buffering is now bounded.
Alignment is no longer critical, if there is resegmentation
it is possible that the start of the placement header is
in the middle of a segment, but most hardware can copy (place)  from
arbitrary offsets and buffer the first part of the segment.
This mechanism is immune to resegmenting middleboxes, if the
preserve the options no problem at all.

The disadvantages are that it is a change to TCP, but minimally
so in that it is just an option. Additionally TCP options
may trigger slow paths through devices, but not requiring an
option on every segment mitigates that at the cost of some
buffering. If a middlebox drops TCP options (due to resegmentation?)
then we are back to an unbounded buffering problem.

While not perfect, this could be considered as a useful
compromise.

	-David

---

From: Randall Stewart <[email-masked]>
Date: Fri, 06 Dec 2002 08:29:23 -0600

Joe:

Just a comment in support of something you said below..

> didn't change between when you checked it/issued the write and when the 

One of the companies I worked for in the past made the above assumption.
All was well and it worked as planned until they were under
tremendous load.. then guess what.. there aligned assumption blew
up .. and so did the application :-0.. in the after-math record
markers were quickly inserted...

R

>> both require that the receiver to support the dual complexity of 
>> handling both aligned and unaligned
>> To me, this seems to have the type of undesirable complexity often 
>> associated with ill advised, ad hoc
>> cases under which the sender will not attempt to align. 
>> now to list here all the implication of this, but the summary is that 
>> this will require
>> right, will not break interoperability with existing implementations.  
>> (However this is
>> a tricky area, and until the work is done, it is not possible to be 
>> sure.)
>>

-- 
Randall R. Stewart
[email-masked] 815-342-5222 (cell phone)

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 6 Dec 2002 12:21:13 -0600

On 12/5/02, Joe Touch wrote:

>
>> Most importantly, DDP does not have to be installed by
>> the system administrator. It *can* run cleanly on top of
>> TCP or SCTP in user mode.
>
>To what end? That's just a waste. The alignment benefit is
>useful only (or 99.9% of the cases) when installed in the
>kernel/enhanced hardware. At that point, use SCTP.
>

It could be kernel/hardware installed at the *other* end,
in which case *that* end receives the performance benefit.

Even if both sides did have software DDP, there is still the
benefit that the Data Source gets to choose the order in
which data is sent.

If you have been asked to deliver a file/object/whatever
that is not stored contiguously on disk this can be a major
benefit to the Data Source.

Of course that is somewhat academic. I doubt very much
whether there will be any software RDDP stacks on the
*server* end of things.

Caitlin Bestler

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Dec 2002 11:24:01 -0700

Caitlin,

Correct me if I'm wrong, but, if DDP/TCP requires changes
to TCP, does it not require kernel changes to be installed
at both ends?  How is installing a modified TCP so different
than installing SCTP?

--brian

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 6 Dec 2002 13:12:55 -0600

On 12/6/02, Brian F. G. Bidulock wrote:

>at both ends?  How is installing a modified TCP so
>different than installing SCTP?
>

There are at least three ways to implement DDP/TCP without
any kernel modifications:

--  raw sockets. User mode code receives and sends complete     
    IP datagrams. You can find many SCTP implementations
    that work exactly this way.
    
--  stream sockets. User mode application sends and receives
    totally standard TCP streams, but makes heavy use of 
    various flags to try to obtain alignment by suppressing
    delay, forcing streams to flush, etc.
    
--  raw device with kernel mode driver interfaces with
    user mode application and with kernel IP stack. But
    kernel IP stack is not modified.
    
    The RNIC is registered as an Ethernet  device with the 
    kernel. Additionally he application is enabled to
    communicate with it bypassing the IP stack. Existing
    applications continue to use the traditional stack.
    RDMA applications use a user mode library that 
    knows how to communicate with the device driver.
    
    Outbound Ethernet frames are accepted either from
    RDMA stacks or from kernel IP stack. Inbound Ethernet
    frames are dispatched based upon L4 peeking.

Use of these bootstrapping approaches is a necessity. If
applications cannot deploy until OS provide RDMA support
then there will be no OS providing RDMA support. After all,
why should they when there aren't any RDMA applications?

Caitlin Bestler

---

From: "Brian F. G. Bidulock" <[email-masked]>
Date: Fri, 6 Dec 2002 16:19:39 -0700

Caitlin,

>     that work exactly this way.

So, that's not different than SCTP.

>     delay, forcing streams to flush, etc.

I understand this does't work to well.

>     frames are dispatched based upon L4 peeking.

Again no different than SCTP.

> why should they when there aren't any RDMA applications?

A few things learned by SCTP implementors:

  Raw sockets suck.
	
  They are non-standard and difficult to get predicted
  behavior visa-vi IP routing.  Implementations based on them
  are largely non-portable (require #ifdef's for every OS).

  Ethernet device drivers are worse.

So, aside from flag tweaking (which I understand doesn't
really work and would require kernel mods in the end anyway),
are you saying there is no difference between DDP/TCP and
SCTP at the endpoint?

--brian