Subject: (IPng 6444) Re: Host Fragmentation

From: James Carlson <[email-masked]>
Date: Fri, 11 Sep 1998 12:52:42 -0400

> I have to doubt whether you have ever programmed common
> networking devices like the 

That's nice.  I'm sure you have my resume and references at hand
before you begin a stupid insult.

> >  It depends.  Many systems have optimized upper and lower interrupt
> >  handlers so that this is not so much an issue.  (Think of BSD's soft
> >  interrupt scheme for IP input; the same can be done for Ethernet input
> >  handlers.)
> 
> Is it being claimed that in common  virtual memory multimode
> interrupt scheduled operating systems, upon receiving an 
> interrupt (in this case a tx done interrupt so that IP input is
> not relevant) data and instruction flow does not become non-local,
> system mode does not often change, page tables never have
> to change and virtual address translation tables do not get
> invalidated?

Again, it depends.  In the system I'm working on at the moment
(BSD-derived), I take one interrupt when the number of free TX buffers
goes above a water mark, and I then fill a large number of them on the
one interrupt.  Taking one interrupt per frame is possible, but silly.

Of course, that's just the Ethernet handler.  The IP stack is separate
and responds much better.  A trace through it shows that with normal
TCP acks, an entire window's worth is dumped into the Ethernet output
queue (the software ifnet queue, not the hardware) in one context
switch.

This supports my contention that IPv4 fragmentation, which is done in
a tight loop in netinet/ip_output.c *after* peeling off a single TCP
segment for transmission in netinet/tcp_output.c, is approximately as
expensive as sending really big frames.

> >  PMTU has nothing to do with this.  Small MTUs do have something to do
> >  with it.
> 
> PMTU procedures would force the jumbogram capable adapter
> to use a small MTU.

Too bad the article you cite doesn't support your conclusion.

> >  If you have a context change problem after reassembly occurs, then
> >  perhaps that's what you should spend some time optimizing.  Such a
> >  context change is not necessary.
> 
> No context change is necessary to get the data into the
> user application?

No, I didn't say that.  I said that if you have a context change
between IPv4 reassembly and TCP input handling -- which would be the
only difference in receiving multiple IPv4 fragments versus receiving
one big packet -- then that context change is silly.

> Hardware based packet switches often have a problem
> with local buffering, but that problem is not confined to
> lost fragments.  Hardware based packet switching is
> an incorrect approach, and kludges to fix the unfixable
> problems associated with hardware based packet
> switching problems should not be standardized.

Huh?  Who was talking about packet switching?  I'm talking about
what's necessary to do IPv4 *REASSEMBLY* versus TCP segmentation.

> The repetitive loss of the same fragment was 
> a result of memory starved systems.  Not a
> problem today.  

Huh?  Port congestion is a fact of life.  Even infinite buffering
won't save you if you're cramming 2Gbps down a 1Gbps pipe.  Losing
packets is a fact of life.  Failing to recover from that gracefully --
which TCP does do, and IP doesn't -- is a severe flaw.

> >  That's why IPv4 fragmentation isn't as good as TCP's segmentation.
> >  Given a choice, I want segmentation, not fragmentation.
> 
> They are both techniques that have their own realms of
> application.  In some cases, IP packets containing TCP
> packets can be fragmented to benefit.

IPv4 fragmentation and reassembly are lame enough that it's not worth
working on them in comparison to working on the link layers
themselves.

> >  I didn't say that IPv4 fragmentation at the host is "good" and that at
> >  the router it's "bad."  I did say that they were equivalent.  If it
> >  amuses you to do IPv4 fragmentation instead of TCP segmentation, then
> >  you can freely do so in your IP output path by having IP lie to TCP
> >  about the MTU.  This involves no context switches and only minimal
> >  overhead.
> 
> The only issue is the IPv6 rule against intermediate system
> fragmentation.  There is no reason for it, it cannot
> even be enforced, and it should be trashed.

You haven't bothered proving your point.

> Only if one refuses to read and understand what I am saying.

Fine.  I obviously made a horrible mistake in reading carefully each
of your posts.  Last reply from me.

> >  	1.  No PMTU, IPv4 fragmentation
> >  	2.  PMTU, no IPv4 fragmentation
> 
> We do the test all the time.  PMTU procedures are a disaster.

Proof?

> Not true in the case of the high performance servers that
> we support.

Does "high performance" measure only dumping bits on the wire or does
it measure application throughput?  These are quite different.

> IP fragmentation requires no context switch.  On multimode,
> virtual memory, interrupt scheduled systems like Unix
> or WindowsNT, a context switch is considerably more than
> a stack change.

Nor does TCP segmentation require a context switch.  Shall I post
lines 78 through 591 of NetBSD sys/netinet/tcp_output.c?  If you have
it at hand, examine carefully what the "sendalot" flag does.

> >  The point, at least for the article you posted, is exactly the effect
> >  of the small Gb Ethernet MTU.  The article said nothing at all about
> >  fragmentation at an intermediate device.  That would be a different
> >  test.
> 
> But it did fairly explicityly indicate lower throughput when not
> using jumbograms.  If jumbogram capability is not available
> on one side, the only way to make any use of jumbograms
> is to use an intermediate fragmentationdevice.

Did you read your own posting?  Here's what it really said:

  [...]
  Large Ethernet frames can improve the efficiency of Gigabit Ethernet
  by reducing the number of frames that must be created and forwarded.
  Packing more data in a single frame can boost performance on
  server-to-server applications, transmitting large amounts of data in
  one stream, such as backups and replication.

  The tests were conducted using a single-processor Intel 400-MHz
  Xeon-based server and four 333-MHz Pentium II clients. An Alteon
  Gigabit Ethernet server switch connected the servers and clients.

  Earlier tests using standard Ethernet frames have shown performance
  from 500Mbps to 600Mbps, said Reza Baghai, NT performance lead
  engineer at Microsoft.

  Alteon is the only Gigabit Ethernet vendor currently supporting large
  Ethernet frames. However, Baghai said Microsoft has had discussions
  with other vendors about using large frame sizes.
  [...]

Did you somehow read fragmentation into that?  If so, then where?  I
see two systems connected back-to-back through a switch supporting
jumbograms and Gb Ethernet.  There's no new data of interest here, and
you've failed to make your point.

-- 
James Carlson, Consulting S/W Engineer  <[email-masked]>
IronBridge Networks / 55 Hayden Avenue  71.246W    Vox:  +1 781 372 8132
Lexington MA  02421-7996 / USA          42.423N    Fax:  +1 781 372 8190
"PPP Design and Debugging" --- http://people.ne.mediaone.net/carlson/ppp
--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------