Subject: [Tsvwg] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

From: "Williams, Jim" <[email-masked]>
Date: Wed, 30 Oct 2002 17:15:37 -0500

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Monday, October 28, 2002 10:28 PM
> To: Williams, Jim
> Cc: [email-masked]; [email-masked]
> Subject: Re: [rddp] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt
>
> [...]
> 
> The elasticity buffer you are citing is an artifact of *one*
> type of implementation with one type of NIC / host
> interface.
> 
> Protocol decisions should still be valid even when DDP
> support is designed into the motherboard, or is part of a
> distributed cache system, etc.
> 
> The point is that each DDP segment is placeable as soon
> as it is received, but the flushing of that placement to
> host memory is subject to a great deal of implementation
> optimization (and hence variation). The only requirement is
> that it be placed *before* the following completion is
> reported to the consumer.
> 
> Because the mechanics of placement is so flexible, and so
> open to optimization, I do not think we should be making
> other protocol design decisions based on the artifacts of
> one way of interfacing the DDP engine and the host.

I agree with your last sentence.  I disagree that it applies
in this case.  Implementation concerns should not drive
protocol definitions if they are an artifact of one
type of implementation.  Implementation concerns
are an important driving factor if they are inherent
in the protocol itself and most reasonable implementations
of that protocol.

When a DDP packet is received, there are some fundamental
things that need to be done before the data can be placed.
These things need to be done independent of the type
of implementation.

  1.  The IP addresses and port numbers must be looked
      up in an associative array to locate the connection
      context.

  2.  The connection context must be referenced.

  3.  PAWS check must be done to determine is packet is valid,
      and sequence number must be checked to insure it is
      within the valid window.  This is done using information
      contained in the context.

  4.  The memory buffer keys must be checked for validity
      and permission, and the physical destination address
      determined.

Some of these involve round trips to large memory structures.
If the number of active connections is large, one can't
necessarily depend on caches.

Is it theoretically possible to do all this and still process
packets at a rate of one every 100ns?  Absolutely.  Is it 
more economical to do this than to add a reasonable amount
of buffer memory and process packets at a rate of one
every 500ns or 1000ns?  My experience would indicate not.
Not even close.

I believe the MPA draft as currently proposed adds considerable
complexity to the protocol, and is considerably more intrusive
on how TCP should be implemented.  It also is quite non-optimal
for implementation that do not attempt to do out of order
data placement.  It therefore requires significant justification
with respect to how it can enable more cost effective 
implementations.

I look forward to hearing more detailed arguments from MPA
proponents describing how MPA will enable more cost
effective implementations.

 - Jim

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Caitlin Bestler <[email-masked]>
Date: Thu, 31 Oct 2002 16:11:07 -0600

On 10/30/02, Williams, Jim wrote:

>
>I agree with your last sentence.  I disagree that it
>applies in this case.  Implementation concerns should not
>drive protocol definitions if they are an artifact of one
>type of implementation.  Implementation concerns are an
>important driving factor if they are inherent in the
>protocol itself and most reasonable implementations of
>that protocol.
>
>  3.  PAWS check must be done to determine is packet is 
>       valid, and sequence number must be checked to insure 
>       it is within the valid window.  This is done using 
>       information contained in the context.
>
>Some of these involve round trips to large memory
>structures. If the number of active connections is large,
>one can't necessarily depend on caches.
>
There is less data here that it might appear, particularly
if the connection is identified through a CAM mechanism.

The connection hash table (presumably in the CAM) is the
largest table: Source IP+Port, Destination IP+Port and the
resulting Context. The number of Destination IPs supported
can be limited to reduce the entry size even further,
especially versus IPv6.

The only required per connection data is the receive window
and the protection domain.

Per STag data: the limits, memory map pointer and pd.

Further, these lookups can execute in parallel (if the STag
is a unique key that leads to a validating PD rather than
the PD + STag forming the key). If the connection hash is
managed by a CAM, parallel execution is not only possible
but natural.

In general, use of memory for lookup has far less impact
that its use as a holding buffer. Holding buffers require
round trips of the entire contents. Table lookups access a
few bytes per packet. Even if the data is held in external
RAM, it would not have to have the same width/frequency as
external RAM being used to round-trip the data would.

>Is it theoretically possible to do all this and still
>process packets at a rate of one every 100ns?  Absolutely.
> Is it more economical to do this than to add a reasonable
>amount of buffer memory and process packets at a rate of
>one every 500ns or 1000ns?  My experience would indicate
>not. Not even close.
>

For many designs you are correct, especially when the design
is optimizing plain TCP as well as RDDP.

The question is relative burden. While MPA markers are ugly,
I do not perceive them as being anything other than a
nuisance. They have to be dealt with, but they will not be
crippling to a NIC that was adopting a use-large-buffer-to-
process-packets-in-order strategy.

On the flip side, the abscence of markers would be a killer
for NICs that wanted to process packets at wire speed
without using external memory as a holding buffer.

>I believe the MPA draft as currently proposed adds
>considerable complexity to the protocol, and is
>considerably more intrusive on how TCP should be
>implemented.  It also is quite non-optimal for
>implementation that do not attempt to do out of order data
>placement.  It therefore requires significant
>justification with respect to how it can enable more cost

I think this is a key area of differing evaluations. MPA is
not pretty, but exactly where do you see it adding
"considerable complexity"?

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg