Subject: (IPng 6457) Re: Host Fragmentation

From: [email-masked]
Date: Mon, 14 Sep 1998 16:09:03 EDT

In a message dated 9/11/98 1:23:44 PM Eastern Daylight Time,
[email-masked] writes:

> > I have to doubt whether you have ever programmed common
>  > networking devices like the 

>  That's nice.  I'm sure you have my resume and references at hand
>  before you begin a stupid insult.

Do I detect defensiveness?  Noting seeming unfamiliarity with
a class of devices is hardly an insult.  Noone can be familiar
with every device.  Carlson need only have pointed out that 
I was mistaken and that he had indeed programmed one
of the devices that I had mentioned. 

BTW, I do put my resume on line ( 
<A HREF="http://members.aol.com/karima4483/resume_c.html">Joachim Martillo</A>
) and provide
a full WAN-LAN level2/level3 switch router for download at my
web site ( <A HREF="http://members.aol.com/Telford001/">Telford Tools, Inc.
</A> ).  If one is going to take part 
in discussions that lead to the definition of an international 
standard protocol, one has an obligation to describe what 
one has done and show what one can do.
  
>  > >  It depends.  Many systems have optimized upper and lower interrupt
>  > >  handlers so that this is not so much an issue.  (Think of BSD's soft
>  > >  interrupt scheme for IP input; the same can be done for Ethernet input
>  > >  handlers.)
 
>  > Is it being claimed that in common  virtual memory multimode
>  > interrupt scheduled operating systems, upon receiving an 
>  > interrupt (in this case a tx done interrupt so that IP input is
>  > not relevant) data and instruction flow does not become non-local,
>  > system mode does not often change, page tables never have
>  > to change and virtual address translation tables do not get
>  > invalidated?
  
>  Again, it depends.  In the system I'm working on at the moment
>  (BSD-derived), I take one interrupt when the number of free TX buffers
>  goes above a water mark, and I then fill a large number of them on the
>  one interrupt.  Taking one interrupt per frame is possible, but silly.

May I ask what the underlying hardware interface device is?
I think the SPYDER-S and SPYDER-T had built-in free
list manipulation, but these devices only used the free buffer
pool for receive.  From the description, I am not sure what
happens in the case of multiple output interfaces, where
several interfaces are disconnected and one can still
transmit.  

If the buffers from the disconnected interface
are not returned to the free pool so that the water mark
is hit, does the system stop transmitting out the
working interface.  The interface actually
sounds rather hideous.  I do not remember such a
mechanism in BSD, but I had to deal with something
like this at Prime when Sachs standardized
hardware communications on OS/x86.  The
watermark interrupt was a software interrupt
that added a layer of complexity, indirection and
performance hit on top of the hardware interrupts.
  
>  Of course, that's just the Ethernet handler.  The IP stack is separate
>  and responds much better.  A trace through it shows that with normal
>  TCP acks, an entire window's worth is dumped into the Ethernet output
>  queue (the software ifnet queue, not the hardware) in one context
>  switch.

>  This supports my contention that IPv4 fragmentation, which is done in
>  a tight loop in netinet/ip_output.c *after* peeling off a single TCP
>  segment for transmission in netinet/tcp_output.c, is approximately as
>  expensive as sending really big frames.

Point?  In the case under discussion, the fragmentation
takes place at an intermediate system off the file/web server.

The idea is to run with TCP windows and max segment
size far larger than a single ether MTU.
  
>  > >  PMTU has nothing to do with this.  Small MTUs do have something to do
>  > >  with it.

>  > PMTU procedures would force the jumbogram capable adapter
>  > to use a small MTU.
  
>  Too bad the article you cite doesn't support your conclusion.

Carlson must go back and reread the article and the discussion.

>  > >  If you have a context change problem after reassembly occurs, then
>  > >  perhaps that's what you should spend some time optimizing.  Such a
>  > >  context change is not necessary.
 
>  > No context change is necessary to get the data into the
>  > user application?
  
>  No, I didn't say that.  I said that if you have a context change
>  between IPv4 reassembly and TCP input handling -- which would be the
>  only difference in receiving multiple IPv4 fragments versus receiving
>  one big packet -- then that context change is silly.

I agree.  I think I have only seen such things in streams
based TCP/IP stacks, and they are silly.
  
>  > Hardware based packet switches often have a problem
>  > with local buffering, but that problem is not confined to
>  > lost fragments.  Hardware based packet switching is
>  > an incorrect approach, and kludges to fix the unfixable
>  > problems associated with hardware based packet
>  > switching problems should not be standardized.

>  Huh?  Who was talking about packet switching?  I'm talking about
>  what's necessary to do IPv4 *REASSEMBLY* versus TCP segmentation.

The issue was not IPv4 REASSEMBLY versus TCP segmentation.
The issue was *host* versus *network* fragmentation.
Network fragmentation takes place in packet switches.
Carlson simply interjected a whole bunch of irrelevant detail.

IPv6 permits host fragmentation but forbids network
fragmentation.  This rule makes no sense and
guarantees bad performance in common situations.
  
>  > The repetitive loss of the same fragment was 
>  > a result of memory starved systems.  Not a
>  > problem today.  
  
>  Huh?  Port congestion is a fact of life.  Even infinite buffering
>  won't save you if you're cramming 2Gbps down a 1Gbps pipe.  Losing
>  packets is a fact of life.  Failing to recover from that gracefully --
>  which TCP does do, and IP doesn't -- is a severe flaw.

Memory starved intermediate systems are not a problem
today.

In the networks where a massive server serves a multitude of
clients interclient communications represents occasional
noise.  I am not sure what the point about IP means as
few applications run raw IP.  If port congestion is due
a statistical fluctuation, having a long queue helps.
If it is endemic, some traffic reengineering must
be performed, to wit, if some large number tcp 
sessions entering the intermediate system through
some large number of input ports are typically
trying to send packets through a single
output port on some intermediate systems,
TCP windowing mechanisms are generally
inadequate to correct the problem, for it
is quite possible that none of the TCP sessions
are making much of a demand on bandwidth.
  
>  > >  That's why IPv4 fragmentation isn't as good as TCP's segmentation.
>  > >  Given a choice, I want segmentation, not fragmentation.

>  > They are both techniques that have their own realms of
>  > application.  In some cases, IP packets containing TCP
>  > packets can be fragmented to benefit.
  
>  IPv4 fragmentation and reassembly are lame enough that it's not worth
>  working on them in comparison to working on the link layers
>  themselves.

Not relevant to the issue of a jumbogram capable server
with clients that are not jumbogram capable.
  
>  > >  I didn't say that IPv4 fragmentation at the host is "good" and that at
>  > >  the router it's "bad."  I did say that they were equivalent.  If it
>  > >  amuses you to do IPv4 fragmentation instead of TCP segmentation, then
>  > >  you can freely do so in your IP output path by having IP lie to TCP
>  > >  about the MTU.  This involves no context switches and only minimal
>  > >  overhead.

>  > The only issue is the IPv6 rule against intermediate system
>  > fragmentation.  There is no reason for it, it cannot
>  > even be enforced, and it should be trashed.
  
>  You haven't bothered proving your point.

The Alteon demonstration proves the point.

>  > Only if one refuses to read and understand what I am saying.
  
>  Fine.  I obviously made a horrible mistake in reading carefully each
>  of your posts.  Last reply from me.
  
>  > >  	1.  No PMTU, IPv4 fragmentation
>  > >  	2.  PMTU, no IPv4 fragmentation

>  > We do the test all the time.  PMTU procedures are a disaster.
  
>  Proof?

PMTU procedures in the Alteon demonstration would have
lead to ~500-600Mbps performance rather than 940Mbps.
I cannot understand how it could be more clear.
  
>  > Not true in the case of the high performance servers that
>  > we support.
  
>  Does "high performance" measure only dumping bits on the wire or does
>  it measure application throughput?  These are quite different.

We run client applications to their maximum during these tests.
The goal is to support as many clients as possible.
  
>  > IP fragmentation requires no context switch.  On multimode,
>  > virtual memory, interrupt scheduled systems like Unix
>  > or WindowsNT, a context switch is considerably more than
>  > a stack change.
  
>  Nor does TCP segmentation require a context switch.  Shall I post
>  lines 78 through 591 of NetBSD sys/netinet/tcp_output.c?  If you have
>  it at hand, examine carefully what the "sendalot" flag does.

While not the main point, there will be far more context switches
at the receiver if max segment is set so that the server sends
out smaller frames.  At 10 to 20 times more context switches
the difference in client performance probably becomes noticeable.

Even if client performance is not an issue, I calculate at least
approximately 20% performance hit in using standard ethernet 
MTU while the Alteon test indicates a 50% performance
hit simply from transmitting smaller frames at the server.
  
>  > >  The point, at least for the article you posted, is exactly the effect
>  > >  of the small Gb Ethernet MTU.  The article said nothing at all about
>  > >  fragmentation at an intermediate device.  That would be a different
>  > >  test.

>  > But it did fairly explicityly indicate lower throughput when not
>  > using jumbograms.  If jumbogram capability is not available
>  > on one side, the only way to make any use of jumbograms
>  > is to use an intermediate fragmentationdevice.
  
>  Did you read your own posting?  Here's what it really said:

>    [...]
>    Large Ethernet frames can improve the efficiency of Gigabit Ethernet
>    by reducing the number of frames that must be created and forwarded.
>    Packing more data in a single frame can boost performance on
>    server-to-server applications, transmitting large amounts of data in
>    one stream, such as backups and replication.

>    The tests were conducted using a single-processor Intel 400-MHz
>    Xeon-based server and four 333-MHz Pentium II clients. An Alteon
>    Gigabit Ethernet server switch connected the servers and clients.
  
>    Earlier tests using standard Ethernet frames have shown performance
>    from 500Mbps to 600Mbps, said Reza Baghai, NT performance lead
>    engineer at Microsoft.
  
>    Alteon is the only Gigabit Ethernet vendor currently supporting large
>    Ethernet frames. However, Baghai said Microsoft has had discussions
>    with other vendors about using large frame sizes.
>    [...]

>  Did you somehow read fragmentation into that?  If so, then where?  I
>  see two systems connected back-to-back through a switch supporting
>  jumbograms and Gb Ethernet.  There's no new data of interest here, and
>  you've failed to make your point.

The setup as I understood it from this article and from the other available
Alteon literature said that performance went from ~500-600Mbps to
940 Mbps per second when an Alteon jumbogram adaptercard is installed
in the server.  The server connects to an Alteon fragmentation
bridge/extruder.
The clients have 100 Mbps Ethercards.  Such setups are pretty much
standard for testing webserver and fileserver performances.

Joachim Martillo
<A HREF="http://members.aol.com/Telford001/">Telford Tools, Inc.</A> 

--------------------------------------------------------------------
IETF IPng Working Group Mailing List
IPng Home Page:                      http://playground.sun.com/ipng
FTP archive:                      ftp://playground.sun.com/pub/ipng
Direct all administrative requests to [email-masked]
--------------------------------------------------------------------