Subject: [IMRG] Network bandwidth vs. system I/O

From: "Loki Jorgenson" <[email-masked]>
Date: Thu, 28 Aug 2003 11:28:29 -0700

Vinay - I tend to agree with Guojun on most of his points including that
"network bandwidth has exceeded system I/O bandwidth", even on 1 Gbps
networks.  While some theoretical claim may be made that bus speeds
technically should be able to handle the rate of packet processing
demanded by 1 Gbps networks, I am tending to believe (without sufficient
solid evidence) that the practical truth is substantially different.

Most "typical", high-end commodity machines (Guojun suggested $500 but I
will settle for anything less than $3000) seem incapable of achieving
full Gbps transfer rates without some significant massaging.  This is
Mathis' "wizard gap" - it is possible to tune them carefully and achieve
better results, possibly even up to full rated capacity, but hardly
typical.  Guojun's call for an accessible, commodity-based solution for
measurement is heartily endorsed by me - any solution isn't a real
solution unless it is "effective".

At a recent Cisco/Intel/IBM webinar on gigE to the desktop,
representatives happily claimed that a typical workstation using 1500
byte MTU could fill the Gbps pipe.  I would refute that claim - some of
the recent measurements made of bandwidth dependence on MTU that ATEAM
have carried out (using probing techniques, as well as other standard
flooding techniques) indicate a serious performance dependence
(presented at the last Joint Techs).  And only a few machines, at 9000
bytes, were able to measure the networks at rated capacity using
probing.  And these are high-end Intel stations running a range of OS
from FreeBSD, to Linux and Windows, intended for a variety of
measurement methodologies.

My conclusion from the data we have (not enough to be really serious but
enough to point in certain directions) is that commodity hardware is
only at the periphery of Gbps performance and it is about to be left
behind as we go to 10, 40 and 100 Gbps.  The hardware performance will
rapidly become the bottleneck and approaches such as Guojun suggests
will be the only feasible ones in the future.

So I would say that Guojun's claim may not "absolutely true" but it is
"effectively true".

On the other hand, any serious proposal for a querying protocol is
entirely welcome - but not likely to be implemented soon enough to avoid
continuing on in this current direction of optimizing probing.

Loki Jorgenson
Apparent Networks
t   604 433 2333 ext 105
m   604 340-2480

-------- Original Message --------
  Date: Fri, 22 Aug 2003 11:50:50 -0500 (CDT)
  From: Vinay Joseph Ribeiro <[email-masked]>
  Subject: [Imrg-dist] Re: [IMRG] New "Available Bandwidth" Measurement
tool
  To: <[email-masked]>
  
  Hi all,
  
  	While Jin has made some excellent points, I would beg to differ
  with him on some issues.
  
  The point that "network bandwidth has exceeded system I/O bandwidth"
is
  misleading. It gives one the impression that machines connected to
Gigabit
  networks cannot read data at 1Gbps (or close to it) which is not quite
  true. The real issue with high-speed NIC cards is that they "coalesce"
  packet arrival interrupts. For example if a NIC receives three
1000byte
  packets spaced 10 micro-sec apart it may send just 1 interrupt to the
  higher layer at the end of the 30 micro-sec interval rather than
bother
  the cpu with 3 interrupts one for each packet. This improves
efficiency
  and actually IMPROVES the total system I/O bandwidth.

_______________________________________________
IMRG mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/imrg

---

From: Constantine Dovrolis <[email-masked]>
Date: Tue, 9 Sep 2003 07:50:16 -0400 (EDT)

Loki, I agree with your reading of the situation. Talking about
saturating a GigE path, with either TCP or UDP, we found out that
using a 32b/33MHz PCI bus is not sufficient to sustain the load
in the I/O system (even though theoretically  32b*33MHz means
1.056Gbps, there is significant overhead in bus arbitration etc).
When we switched to boxes (all of them < $2500) with 64b/66MHz PCI,
the problem went away.

Constantinos

--------------------------------------------------------------
Constantinos Dovrolis | 218 GCATT | 404-385-4205
Assistant Professor | Networking and Telecommunications Group
College of Computing | Georgia Institute of Technology
[email-masked]
http://www.cc.gatech.edu/fac/Constantinos.Dovrolis/