Subject: (IPng 6439) Re: Host Fragmentation

From: James Carlson <[email-masked]>
Date: Fri, 11 Sep 1998 08:03:14 -0400

> The analysis of transmit DMA is not complete.  In many environments
> either the DMA device polls, which costs transactions to memory or the
> DMA device must be told to start when a new message is inserted into a
> previously empty transmit queue.  Either the transmit routine kicks
> the DMA device everytime a new message is inserted or there is some
> (potentially complex logic) to determine whether to kick the DMA
> device.  I recommend not using the DMA polling.  As for DMA transmit
> start logic, however implemented, the less frequently (i.e. the bigger
> the outgoing messages) the better.  For a lot of common busses
> (e.g. PCI) the transactions associated with DMA start are costly.

No devices that I've ever used have a hardware transmit or receive
queue of length one.  Perhaps your experience is different.  Again, my
previous post gave details on why transmitting packets is much easier
than receiving.  Your post contains no such details.

> While it is not necessarily true for turnkey packet switches and file
> servers, standard Unix and WindowsNT file servers receive network
> interrupts at a rate that is monotonically increasing with the number
> of packets sent and received.

True.  That's why having fewer, bigger packets in general is helpful,
and that's why your original article showed what it did.  This is
nothing new.  And it proves nothing at all about fragmentation one way
or the other.

> Interrupts are bad.  Instruction caches have to be invalidated.  Data
> and instruction flow becomes non-local.  System mode may have to
> change.  Page tables may have to be changed.  Look-aside tables (or
> whatever they are called on the current system) may be invalidated.
> System performance goes down.

It depends.  Many systems have optimized upper and lower interrupt
handlers so that this is not so much an issue.  (Think of BSD's soft
interrupt scheme for IP input; the same can be done for Ethernet input
handlers.)

> But as my partner points out, the real problem of receive and transmit
> performance is running the protocol stack and context switching.  To
> get the best performance PMTU procedures should never be used.

PMTU has nothing to do with this.  Small MTUs do have something to do
with it.

> If jumbograms are used on the server, the server runs the
> stack/context change once for each jumbogram whereas in the PMTU case
> the stack/context change will be run for each of the smaller
> packets.

That's not necessarily so.  When your TCP window opens and you can
send a number of packets, you can certainly do so with no context
changes whatsoever.  If multiple short trips over the IP output path
cause trouble, then perhaps this is what needs to be optimized.

> At the host that receives fragments, with a reasonable driver and
> reasonable memory management, with a correct fragmentation reassembly
> implementation, there is a performance win in gathering all the
> fragments together for one upstack/context change call versus one
> upstack/context change call for each individual packet the size of one
> of the packet fragments.

If you have a context change problem after reassembly occurs, then
perhaps that's what you should spend some time optimizing.  Such a
context change is not necessary.

> If PMTU procedures should never be used, then in the context of IPv6,
> the question is the following.
> 
> Why is fragmentation at the host good while fragmentation at the
> router is bad?

IPv4 fragmentation does not have TCP's congestion management
mechanism.  One lost fragment causes the remaining mess to sit in
local buffers for a long period of time before being discarded, and
the entire packet has to be resent.  TCP, on the other hand, will
quickly resend the one missing piece and continue on its way.

That's why IPv4 fragmentation isn't as good as TCP's segmentation.
Given a choice, I want segmentation, not fragmentation.

I didn't say that IPv4 fragmentation at the host is "good" and that at
the router it's "bad."  I did say that they were equivalent.  If it
amuses you to do IPv4 fragmentation instead of TCP segmentation, then
you can freely do so in your IP output path by having IP lie to TCP
about the MTU.  This involves no context switches and only minimal
overhead.

> There is no logical answer to the question.  It is possible to build
> router/extruders that work at full bandwidth and completely off-load
> all the transmit overhead (described above) associated with host
> fragmentation.  Why forbid such off-loading and why pay attention to
> such a rule as an IPv6 router could certainly fragment a packet, and
> there would be no way to determine that the packet was not fragmented
> at the host?

Once again, your original posting had nothing whatsoever to say about
the relative merits of fragmenting in a router versus fragmenting in a
host.

> >  (In fact, I strongly suspect that, given the skimpy data and
> >  nonexistent analysis, all this test *really* proved was that either NT
> >  doesn't have decent TCP header prediction, or that it doesn't have
> >  good pcb hashing, or that it has terrible context-switching
> >  properties.  Or maybe it just doesn't have the TCP window scaling
> >  option.)
> 
> A red-herring.  Even if all the costs of context-switching,
> PCB look-up or pcb hashing were low, why would one
> want to do such things many times when they could
> be done once?

Because, if they are low, then the benefits of TCP's far more elegent
reassembly algorithm can be used.

> The test supported an observation that I have seen in the optimization
> of massive file and web server performance.

Supporting a result is not the same as duplicating it.  The article
that you posted talked about Gb Ethernet transfers using large packets
between peers.  This has little to do with web server design or
performance.

>  There is practically
> nothing worse for the server than transmitting frames less than the
> maximum size that the server can transmit.  I have seen some fairly
> dramatic increases in performance in merely swapping a 100 Mbps
> ethernet interface with an FDDI interface on DECUNIX or DG/UX.

This proves only that large packets are better.  We're back where we
started.  It proves nothing about fragmentation.

>  PMTU
> is a disaster for massive file and web server performance while
> off-loading the cost of fragmentation is definitely a plus.

To prove this, as this article most certainly does NOT do, you'll need
to use this set-up:

	Server machine <--large-MTU--> router <--small-MTU--> peer

Over this, run a long-term test with two cases:

	1.  No PMTU, IPv4 fragmentation
	2.  PMTU, no IPv4 fragmentation

Then determine what happens when the "small-MTU" link -- which happens
to correspond with the general Internet in most cases -- experiences
congestion.  (This can be modeled as random packet discards.)

The result, as has been shown many times, is that for trivial cases,
where there is absolutely no packet loss and no congestion, these
cases are approximately the same (with case 1 being a percent or so
faster; your mileage will vary).  These are the kinds of tests usually
run for those trade rag articles.  If, however, any congestion at all
is encountered, case 2 wins every time by a country mile.

> >  If (1) and (2) above cannot be combined for a net gain in performance,
> >  then the "server" in (1) is lame and should be replaced.
> 
> The exact opposite is true because the stack/context change is run for
> each smaller packet.

If you require a context change during IP fragmentation, then I'd say
your stack needs a serious overhaul.  If the stack changes cause
noticable degradation, then in-line the code or special-case it.  I
stand by the earlier comment.  If there's any measurable difference,
then this server should be replaced.  There's no good reason a router
should be able to do IPv4 fragmentation any faster than a good host
can do it.

> >  That's exactly the point.  The article you posted shows the benefits
> >  of using large MTUs on media with low error rates.  It tells nothing
> >  about using fragmentation.  They never even mentioned it.  So how can
> >  fragmentation itself be counted on as a panacea here?
> 
> Actually, it shows the effect of packet size on server performance.

No, it shows the effect of MTU on performance between peers.  There's
a big difference here.

> The point is not the medium.  From the standpoint of impinging server
> performance, if it is possible to off-load full bandwidth fragmentation (as
> it is) it is wrong for the server not to transmit the maximum size
> packets that can then be fragmented at an intermediate device.

The point, at least for the article you posted, is exactly the effect
of the small Gb Ethernet MTU.  The article said nothing at all about
fragmentation at an intermediate device.  That would be a different
test.

-- 
James Carlson, Consulting S/W Engineer  <[email-masked]>
IronBridge Networks / 55 Hayden Avenue  71.246W    Vox:  +1 781 372 8132
Lexington MA  02421-7996 / USA          42.423N    Fax:  +1 781 372 8190
"PPP Design and Debugging" --- http://people.ne.mediaone.net/carlson/ppp
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------