Subject: [Tsvwg] RE:I-D ACTION:draft-williams-iwarp-ift-00.txt

From: "Williams, Jim" <[email-masked]>
Date: Mon, 28 Oct 2002 12:27:25 -0500

> -----Original Message-----
> From: Caitlin Bestler [mailto:[email-masked]]
> Sent: Friday, October 25, 2002 12:10 PM
> To: [email-masked]; [email-masked]
> Cc: Williams, Jim
> Subject: Re: [rddp][Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt
> 
> 
> On 10/25/02, Williams, Jim wrote:
> 
> 
> >
> >This is really the critical core issue we need to discuss 
> >in more depth.
> >
> >My views tend to be as follows.
> >
> >  A.  I think you are underestimating the amount of memory
> >      required for elasticity buffering, and overestimating
> >      the memory required for out of order reassembly.
> >
> >  B.  Once the cost is incurred of adding external (off 
> >       ASIC) memory of sufficient bandwidth for elasticity, 
> >       the incremental cost of additional megabytes is 
> >       quite low.
> 
> True. But particularly at the lower bit rates, on-chip
> memory could easily be adequate for re-assembling split DDP
> segments and buffering host bus access.

I believe an important objective in the protocol design
should be avoiding the need to re-assemble split
DDP segments.  This is an important motivating factor for
a separate header CRC.

The idea is that the first part of the split DDP segment can
be placed without the need to reassemble it.  When the
second part of the split segment arrives, payload CRC
computation resumes where it left off on the first part.
All that needs to be saved in the interim are the header
info and the CRC seed.

> Also consider that during the life of the protocol, the
> scope of on-chip memory is only going to improve.

So will the required memory as bandwidths increase.
It's not clear which will increase faster, but I think
I put my money on network bandwidths.

> >  D.  A product design needs to set performance goals at 
> >       two (at least) different traffic mixes, large
> >      messages, and small messages.  An accelerated NIC
> >      which is able to support line rate with small 
> >       messages is seriously over designed and non cost 
> >       effective.
> >
> >      Supporting line rate at large and medium size 
> >       messages is reasonable, and requires significantly 
> >       lower peak packet processing rates.
> >
> >      The result of this is that for small messages, one
> >      needs to rely on elasticity buffering and end to 
> >      end flow control to avoid large scale dropping of
> >      incoming packets.  With insufficient buffering,
> >      the supported rate for small messages will be 
> >      terrible due to constant dropped packets and 
> >      retransmission timeouts. 
> >
> 
> That is a characteristic of a general TCP optimizer. I do
> not believe that it applies to framing DDP segments. The
> point of the MPA proposal is that it allows each DDP segment
> to be processed on it own as it arrives, and further that in
> virtually every case there will be exactly one DDP segment
> per TCP segment / IP datagram / Ethernet Frame.
>
> Under that scenario the size of the tagged or untagged
> message is totally irrelevant, each segment is placed
> as it arrives.

The size is relevant because it is inversely proportional
to the segment arrival rate. On a 10Gb link, 100 byte
DDP segments can arrive every 100ns (approx).
1000 byte DDP segments can arrive every microsecond.
Doing a direct placement every 100ns is
impractical.  It's probably impossible with
current host bus designs.

Therefore the segments must be held in the 
elasticity buffer, and end to end flow control
must be used to avoid overflowing the buffer.
It's this requirement that drives the size of
the needed buffering.

> [...]
> 
> Caitlin Bestler
> http://asomi.com/CaitlinBestler/
> 
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: Caitlin Bestler <[email-masked]>
Date: Mon, 28 Oct 2002 21:28:15 -0600

On 10/28/02, Williams, Jim wrote:

>The idea is that the first part of the split DDP segment
>can be placed without the need to reassemble it.  When the
Good point. It also clarifies the objective for keeping the
SCTP mapping equivalent. If the SCTP mapping does not allow
fragmentation of chunks then there is no need to protect
just the header. Otherwise, the DDP header and the payload
would have to be two different chunks.

>> 
>> That is a characteristic of a general TCP optimizer. I
>> do not believe that it applies to framing DDP segments.
>> The point of the MPA proposal is that it allows each DDP
>> segment to be processed on it own as it arrives, and
>> further that in virtually every case there will be
>> exactly one DDP segment per TCP segment / IP datagram /
>> Ethernet Frame.
>>

As far as the RDDP engine is concerned, data can be placed
immediately as each segment is processed. The interactions
with the host bus controller is a detail of cache
management.

The elasticity buffer you are citing is an artifact of *one*
type of implementation with one type of NIC / host
interface.

Protocol decisions should still be valid even when DDP
support is designed into the motherboard, or is part of a
distributed cache system, etc.

The point is that each DDP segment is placeable as soon
as it is received, but the flushing of that placement to
host memory is subject to a great deal of implementation
optimization (and hence variation). The only requirement is
that it be placed *before* the following completion is
reported to the consumer.

Because the mechanics of placement is so flexible, and so
open to optimization, I do not think we should be making
other protocol design decisions based on the artifacts of
one way of interfacing the DDP engine and the host.

Caitlin Bestler