Subject: RE: [Tsvwg] I-D ACTION:draft-williams-iwarp-ift-00.txt

From: "Williams, Jim" <[email-masked]>
Date: Fri, 25 Oct 2002 11:16:49 -0400

> -----Original Message-----
> From: Culley, Paul [mailto:[email-masked]]
> 
> Interestingly enough, during our framing discussions, both 
> the added header CRC and Markers were seriously discussed 
> before we came to our conclusions.  We had several NIC 
> vendors present who presented private analysis as well as 
> (too many) opinions based on various potential NIC models.  
> We came to several conclusions:
> 
> 1) You are right that packet processing rates require an 
> elasticity buffer.
> 
> 2) When many connections are running at one time, the NIC 
> memory needed for dealing with the  out-of-order packets 
> scales by the number of connections.  We intend to support 
> large numbers of connections (100s to 10000s).  The memory 
> needed rapidly goes out of range of what is needed for 
> elasticity.  The amount of memory also scales with latency 
> bandwidth product, at 1-10 gigabit this gets nasty very fast. 
> Thus out of order processing and markers to absolutely 
> identify them...

This is really the critical core issue we need to discuss 
in more depth.

My views tend to be as follows.

  A.  I think you are underestimating the amount of memory
      required for elasticity buffering, and overestimating
      the memory required for out of order reassembly.

  B.  Once the cost is incurred of adding external (off ASIC)
      memory of sufficient bandwidth for elasticity, the
      incremental cost of additional megabytes is quite
      low.

  C.  I question whether memory required for out of order
      reassembly scales with number of connections.  On a
      particular connection, the required memory should 
      never be larger than the bandwidth delay product
      of that connection.  The more active connections
      there are, the less bandwidth is available for each
      connection.

  D.  A product design needs to set performance goals at two
      (at least) different traffic mixes, large
      messages, and small messages.  An accelerated NIC
      which is able to support line rate with small messages
      is seriously over designed and non cost effective.
      Supporting line rate at large and medium size messages
      is reasonable, and requires significantly lower peak
      packet processing rates.

      The result of this is that for small messages, one
      needs to rely on elasticity buffering and end to 
      end flow control to avoid large scale dropping of
      incoming packets.  With insufficient buffering,
      the supported rate for small messages will be 
      terrible due to constant dropped packets and 
      retransmission timeouts. 

Unfortunately, at the moment I don't have quantitative
analysis sufficiently clear and documented to share.
I will try to work on that, but it is tricky due the
need to make a number of somewhat arbitrary assumptions.

In the mean time it would be helpful for the RDMAC to share
what ever quantitative analysis they have done to reach 
their conclusions.

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Julian Satran" <[email-masked]>
Date: Fri, 25 Oct 2002 17:46:51 +0200

Jim,

Do you have some data about the cost ratio for your point B (initial cost 
versus cost/MB)?

And I fully agree with C. It amazes me how often I hear this memory 
scaling and connections "argument".
It is probably true in a VIA-user-space setting with every connection 
ending in a separate address space and having to be designed for the link 
max-speed and not the real connection usage but certainly not true in a 
NIC where buffers are anonymous in a common pool.

BTW the above is also the strongest argument against long term real 
(pinned) buffers for user-space communication and it has been made ages 
ago.

Julo

"Williams, Jim" <[email-masked]>
Sent by: [email-masked]
25/10/02 17:16
 
        To:     [email-masked], [email-masked]
        cc: 
        Subject:        [rddp] RE: [Tsvwg] I-D 
ACTION:draft-williams-iwarp-ift-00.txt

_______________________________________________
rddp mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/rddp

---

From: Caitlin Bestler <[email-masked]>
Date: Fri, 25 Oct 2002 11:10:20 -0500

On 10/25/02, Williams, Jim wrote:

>  B.  Once the cost is incurred of adding external (off 
>       ASIC) memory of sufficient bandwidth for elasticity, 
>       the incremental cost of additional megabytes is 
>       quite low.

True. But particularly at the lower bit rates, on-chip
memory could easily be adequate for re-assembling split DDP
segments and buffering host bus access.

Also consider that during the life of the protocol, the
scope of on-chip memory is only going to improve.
>      there are, the less bandwidth is available for each

You still have the split round-up effect. The buffering for
each connection must be rounded up due to allocation
granularity and bookkeeping overhead.

An algorithm that uses a large allocation granularity would
have a dramatic increase in the total memory required when
the number of connections rose. There are alternate
algorithms that will reduce this, however.

>  D.  A product design needs to set performance goals at 
>       two (at least) different traffic mixes, large
>      which is able to support line rate with small 
>       messages is seriously over designed and non cost 
>       effective.
>
>      Supporting line rate at large and medium size 
>       messages is reasonable, and requires significantly 
>       lower peak packet processing rates.
>

That is a characteristic of a general TCP optimizer. I do
not believe that it applies to framing DDP segments. The
point of the MPA proposal is that it allows each DDP segment
to be processed on it own as it arrives, and further that in
virtually every case there will be exactly one DDP segment
per TCP segment / IP datagram / Ethernet Frame.

Under that scenario the size of the tagged or untagged
message is totally irrelevant, each segment is placed
as it arrives. The size and frequency of untagged messages
would have a slight affect on how often completions were
signalled. But it is hard to imagine an rddp scenario where
the host memory interactions related to receive queue
consumptions and completion queue posting would be
signifigant compared to the raw data placement.

Caitlin Bestler
http://asomi.com/CaitlinBestler/
_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg