Subject: (IPng 6433) Re: Host Fragmentation

From: James Carlson <[email-masked]>
Date: Thu, 10 Sep 1998 15:13:11 -0400

> In most networked systems (but not the VLAN Router I admit),
> transmitting one jumboframe costs a lot less than transmitting
> 10 small frames.

Ok, prove it.

My claim is that the costs are virtually the same for transmission of
small and large frames.  In order to set up one large block of data,
the transmitter must set up a list of buffer descriptors for the
hardware DMA engine, corresponding to the memory buffers to be sent.
In order to set up several small blocks of data, the transmitter must
also set up a list of buffer descriptors corresponding to the memory
buffers to be sent.  In the best case for jumbograms -- if you assume
that internal network buffers are at least as large as the MTU -- this
means that you go from two memory transactions (write out length +
pointer for one buffer) all the way up to twenty memory transactions
(same thing 10 times).  In reasonable implementations, buffers will be
smaller, and the difference between the two will be correspondingly
smaller.  For 2KB buffers and below, there is no difference at all.

Additionally, the overhead of the DMA itself is the same in both cases
-- it's the same number of bytes.  And the number of memory
transactions is large compared to the CPU work, but is the same in
both cases; perhaps 16K reads for each 64KB block of data.

The rest of the transmit overhead processing is a wash.

On the receive side, things are quite a bit different.  Every received
frame must have its Ethertype checked, its IP header validated, and
then run through the per-peer reassembly list.  Completed frames must
be reassembled into a single IP frame (note that we've been talking
about *IP* reassembly, not TCP), its IP destination address matched
against the local address list, its TCP header validated, its TCP pcb
located, TCP header prediction/reassembly run, and then delivered to
the application.

On a good day, you might be able to do the Ethernet and IP checks in
30 instructions plus about 8 memory transactions PER PACKET.  So, the
overhead is about 4:1 compared with the transmission overhead for the
same number of packets.

Thus, I claim that transmission is cheap in comparison with reception.

> With reasonable receive message buffer management and
> reasonable receive hardware, it makes little difference
> whether the buffers of an IP packet are filled from
> one received IP packet or from multiple fragments
> of the IP packet.

That's correct in terms of the hardware, but wrong in terms of
performance.  The hardware for data reception not an issue -- it's the
software to handle the IP header's that's the performance-hampering
issue.

(In fact, I strongly suspect that, given the skimpy data and
nonexistent analysis, all this test *really* proved was that either NT
doesn't have decent TCP header prediction, or that it doesn't have
good pcb hashing, or that it has terrible context-switching
properties.  Or maybe it just doesn't have the TCP window scaling
option.)

> Thus in the current situation to get the highest performance
> network service it makes sense when clients cannot
> receive jumbograms (the common situation),
> 
> 1) for the server to transmit jumbograms,
> 
> 2) to interpose a device capable of rapid fragmentation and
> 
> 3) for the client to receive the fragments and reassemble
> the IP packet from fragments of its MRU size.

If (1) and (2) above cannot be combined for a net gain in performance,
then the "server" in (1) is lame and should be replaced.

> >  	- Larger MTUs are better than smaller ones if throughput is
> >  	  your goal (certainly not a surprise)
> 
> But then one should wonder how to optimize network performance
> when the receiver MRU is much lower than the transmitter MTU.

That's exactly the point.  The article you posted shows the benefits
of using large MTUs on media with low error rates.  It tells nothing
about using fragmentation.  They never even mentioned it.  So how can
fragmentation itself be counted on as a panacea here?

> Whether the server is NT or not is irrelevant. Lots of Unix network
> drivers could be better.  But one can reasonably observe that
> IPv6 may force suboptimal network performance until jumbogram
> capable receivers are installed in network clients.  Forcing 
> suboptimal network performance indicates a failure or
> inadequacy of IPv6.  Good engineering requires modifying
> IPv6 to correct this inadequacy.

Since smallish MTUs on Gb Ethernet isn't IPv6's fault, and since Gb
Ethernet is not the only medium that IPv6 will run over, there's
nothing to be fixed here.  IPv6 neither knows nor cares about this
oddity.  Perhaps it's an issue for IEEE 802.3z ...

-- 
James Carlson, Consulting S/W Engineer  <[email-masked]>
IronBridge Networks / 55 Hayden Avenue  71.246W    Vox:  +1 781 372 8132
Lexington MA  02421-7996 / USA          42.423N    Fax:  +1 781 372 8190
"PPP Design and Debugging" --- http://people.ne.mediaone.net/carlson/ppp

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------