Subject: [Tsvwg] Re: [e2e] Are you interested in TOEs and related issues

From: Sunay Tripathi <[email-masked]>
Date: Wed, 3 Mar 2004 20:55:58 -0800 (PST)

Craig,

> Well, the interesting thing is we went down this path in the early 1980s
> and found that TOEs didn't work.
> 
> If I try to distill all that we learned in the 1980s into one question, I
> come out with:
> 
>     In the 1980s we discovered that communicating with a TOE over a bus
>     about a TCP connection was as expensive or more expensive than
>     simply handling the TCP connection in the main processor.  What about
>     today's TOE designs makes them different?

I don't have too much history (1980s predates my career) but there were
some attempts in mid 90s to do the same and compared to that, the
following reasons seem to be the driving force:

1) Extra processor(s) buried in the TOE for networking processing which is
   hidden from the kernel and leaves the host CPU to do more application
   related work. Saves the cost of licences for application which take
   number of CPU into account (oracle is one such application cited).
2) On low end (1-2 CPU) x86 based machines, cost of adding a processor
   is much higher than adding a TOE (I personally haven't verified this).
3) For the up and coming 10Gb NICs, TOE will help saturate the link. Some
   vendors assert that TOE will be required to support 10Gb NICs.
4) Performance reasons. Just the LSO aspect of TOE (sending large chunks of
   data and letting the TOE split it up in mss size pieces) and ack
   coalescing gives a pretty good boost (our own prototypes indicates that
   this is true). The gains are by optimizing data movement and not by
   offloading protocol processing.
5) TOE is necessary for RDMA, iSCSI etc. for layering reasons. I am not
   involved with RDMA so someone who is an expert can probably comment on
   this part.
6) TOE based NIC are already making pretty good headway in embedded space.
   The technology is already maturing so why not use it in broader market.

Note that the above claims are in no particular order of importance and made
my TOE vendors in general. Of these, I personally do agree with 1 and 4 but
that iteself doesn't mean that TOE will make it in general purpose networking.

It would be interesting to see if you and others in the list agree or
disagree with these claims.

Thanks,
Sunay

-- 

Sunay Tripathi
Senior Staff Engineer,
Solaris Kernel Networking,
Sun MicroSystems Inc.

email: [email-masked]		 Phone:	650-786-6007 (W)

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "Leonid Grossman" <[email-masked]>
Date: Wed, 3 Mar 2004 22:29:22 -0800

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Sunay Tripathi
> Sent: Wednesday, March 03, 2004 8:56 PM
> To: [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: [Tsvwg] Re: [e2e] Are you interested in TOEs and 
> related issues
> 
> 
> Craig,
> 
> > Well, the interesting thing is we went down this path in the early 
> > 1980s and found that TOEs didn't work.
> > 
> > If I try to distill all that we learned in the 1980s into one 
> > question, I come out with:
> > 
> I don't have too much history (1980s predates my career) but 
> there were some attempts in mid 90s to do the same and 
> compared to that, the following reasons seem to be the driving force:
> 

TOE will bring CPU utilization down of course, but 10GbE NIC will
saturate the link with or without a TOE - much like GbE NICs do it now.

>    this is true). 

Just curious - in your testing, how much CPU is freed up by the ack
coalescing?

>The gains are by optimizing data movement and not by

LSO is a 'stateless' offload (much like checksum offload) so one doesn't
need a TOE to support it; 
most modern NICs (including our 10GbE Adapter) do LSO. Also, for normal
frames LSO has quite dramatic effect at 10GbE rates, but for Jumbo
frames performance gain diminishes to ~7%.

> Note that the above claims are in no particular order of 
> importance and made my TOE vendors in general. Of these, I 
> personally do agree with 1 and 4 but that iteself doesn't 
> mean that TOE will make it in general purpose networking.
> 
> It would be interesting to see if you and others in the list 
> agree or disagree with these claims.
> 
> Thanks,
> Sunay
> 
> -- 
> 
> Sunay Tripathi
> Senior Staff Engineer,

Regards, 
Leonid Grossman
www.s2io.com

---

From: Sunay Tripathi <[email-masked]>
Date: Wed, 3 Mar 2004 23:10:06 -0800 (PST)

<SNIP>
> saturate the link with or without a TOE - much like GbE NICs do it now.

I think the TOE vendors meant that single CPU won't be able to saturate
the 10Gb NIC on a more meaningful workloads than just a simple throughput
tests. Even a 2 CPU machine which wants to do some work also might also
get constrained.

> Just curious - in your testing, how much CPU is freed up by the ack

We actually always try to run the CPUs to saturation (keep increasing
the load) and we can get a double digit percentage boost on a web
like workload (multiple simultaneous connections). I didn't verify
exactly where the gains were coming from but my feeling was that
more efficient data movement and better locality was contributing to
bulk of the gains.

> need a TOE to support it; 

It depends. If the kernel does a write based on open window, then yes
you are right. But if you have full TOE and you pass large chunks of
data to TOE (does wonder when sendfile/sendfilev is in play) and let
TOE figure out when to send it, the gains increase. In the later case,
TOE needs teh TCP state.

> frames performance gain diminishes to ~7%.

Do you have any insight on why you would see a degradation with Jumbo
frames? As long as you are stateless, the jumbograms should show
no degradation. 

Thanks,
Sunay

---

From: stanislav shalunov <[email-masked]>
Date: 05 Mar 2004 01:33:53 -0500

>    number of CPU into account (oracle is one such application cited).

One would hope TCP design would not be guided by Oracle licensing quirks.

>    is much higher than adding a TOE (I personally haven't verified this).

It's not obvious why this should necessarily be the case, given that
it is likely that there will continue to be quite a bit more
general-purpose CPUs made than TCP offload engines.

>    vendors assert that TOE will be required to support 10Gb NICs.

Right now, one can do almost 8Gb/s with a single TCP stream over 10GE
(let's say 5 or 6Gb/s with more common hardware).  The limiting factor
is currently host bus speed.  There's nothing (except for compression
on the bus side, but it should be clear we're not going there) that an
offload engine can do about host bus speed.  By the time host busses
faster than 10Gb/s are commonly available, pretty routine x86 box
should handle 10GE saturation.

>    offloading protocol processing.

Here I would agree with the unnamed TOE vendor whom you're
paraphrasing (and with Jerry Chu's comments in this thread).  Ethernet
frame size (1500 bytes or even 9kB) is very small; we'd be in a better
world if we could specify the MTU in units of time (the number of CRC
bits would have to scale up as well, of course).  TCP offload engines
could be a kludge that would help to work around this deficiency in
Ethernet.

Note that since even at 10Gb/s one CPU is enough to saturate the link
with 9kB packets, the need for this work-around is not at all
pressing.  Given the potential harmful effects (undetected errors on
the host bus, difficulty in patching stale or buggy TCP code, etc.),
one would probably be better served by concentrating on the deployment
of jumbo frames.  The investment to support jumbo frames has largely
already been made, so why not extract all we can from it first?

-- 
Stanislav Shalunov		http://www.internet2.edu/~shalunov/

---

From: Jeremy Harris <[email-masked]>
Date: Fri, 05 Mar 2004 17:02:37 +0000

> general-purpose CPUs made than TCP offload engines.

It's because cache-coherency is expensive.  x86 boxes have traditionally
avoided playing far into this space.  Looked at from that angle,
TOE is towards the far end of the ncNUMA scale, and coherency
is managed by software.

- Jeremy

---

From: Jeff Chase <[email-masked]>
Date: Thu, 04 Mar 2004 09:50:53 -0800

The argument for offload often takes the premise that host CPU is a
bottleneck at high network speeds, which is often true.  If we can
take that as given, then it is easy to capture the effects of
transport offload on throughput using a back-of-napkin analysis.

One issue is the ratio of host speed to NIC speed for protocol
processing.  The benefit from offload is bounded by the inverse of
that ratio.  In the 1980s, the offload NICs took longer than the host
to process the protocol, and the wire was fast enough to saturate
either of them.  In that case, and only in that case, offload may
actually reduce throughput rather than increase it.

Today, technology improvements and the role of offload in copy
reduction (e.g., RDMA) help to offset this concern.  Jeff Mogul had a
nice paper in the last HotOS called 'TCP offload is a dumb idea whose
time has come' dealing with this point.

One can also draw some interesting conclusions about the maximum
benefit of offload as a function of the ratio of host speed to wire
speed, and of the compute/communicate ratio of the application.
Crudely, the maximum speedup from offload is bounded by the inverse of
both ratios.

Much of the argument about whether offload makes sense boils down to
disagreement about how trends are affecting these various ratios, or
confusion about how these ratios affect potential speedup.

My student Piyush Shivam and I wrote up a 6-page paper with a simple
analysis for the NICELI workshop at SIGCOMM last year, called "On the
Elusive Benefits of Protocol Offload".  It is just Amdahl's Law and
some high-school math.  If you are interested, you can find it at:

http://issg.cs.duke.edu/publications/niceli03.pdf

Regards,
Jeff

http://www.cs.duke.edu/~chase

Craig wrote:
 >...

---

From: Craig Partridge <[email-masked]>
Date: Sun, 07 Mar 2004 18:38:54 -0500

In message <[email-masked]>, Sunay Tripathi w

>   number of CPU into account (oracle is one such application cited).

Right, and the problem is that if the TOE is very sophisticated, you can't
hide it from the host CPU.  Remember when the application says "write",
on a socket, some magic has to occur to move a buffer from application space
to the right TCP connection -- and past experience says that magic has
90% of your non-memory moving costs...

>   vendors assert that TOE will be required to support 10Gb NICs.

For any speed X, where X is viewed as large, there will be people claiming
they have just the innovation needed to achieve that speed.

>   offloading protocol processing.

Agreed -- plenty of evidence that if you don't do protocol processing but
just help moving memory, you win.

>   involved with RDMA so someone who is an expert can probably comment on

Certainly for memory moving it is.

>   The technology is already maturing so why not use it in broader market.

In the embedded space you've got a different cost-benefit tradeoff.  The
embedded processor may often have a single application...

Craig

---

From: Sunay Tripathi <[email-masked]>
Date: Tue, 9 Mar 2004 16:58:45 -0800 (PST)

> One would hope TCP design would not be guided by Oracle licensing quirks.

I don't think we are designing TCP here. We are just discussing an
implementation of TCP and obviously costs do play a factor in any
implementation. 

> should handle 10GE saturation.

We have host busses today which can handle a giga bytes plus on the
back plane. Infiniband based machines are already hitting and there
are other technologies (that I can't talk about yet) which will make
the host bus issues irrelevant. 

> already been made, so why not extract all we can from it first?

Sure, jumbo frames do deserve attention. Its being discussed right now
in IETF but I am not sure 9k frames would be enough to saturate the
10Gb NIC till the processor speeds also scale significantly. Note that
its not just the ability to saturate the link but you also need to
have some CPU free to do real work as well.

Cheers,
Sunay

---

From: Sunay Tripathi <[email-masked]>
Date: Tue, 9 Mar 2004 18:59:02 -0800 (PST)

> 90% of your non-memory moving costs...

The question is whether you want to hide the TOE from the application or
kernel as well? The current implementations override the S/W stack and 
put their own hooks in sockets so that TOE is kind of hidden from application
as well as the kernel (or atleast the kernel can do nothing about it).
I am pretty sure that this won't fly. You can hide the TOE from the
application but the kernel needs to control it. And once the kernel
controls the TOE, the magic becomes easy (or not expensive atleast).
My current estimate is that I probably spend 10% more instructions in
connection setup/teardown but in return I have an extra processing
element. And to sweeten the deal, I am moving data in bigger chunks
and saving on the memory costs also. 

> they have just the innovation needed to achieve that speed.

Agreed totally :)

Thanks,
Sunay

---

From: Michael Krause <[email-masked]>
Date: Tue, 09 Mar 2004 22:53:34 -0800

>already been made, so why not extract all we can from it first?

Jumbo frames are not a panacea:

- Workloads use a variety of message sizes with many messages being 
significantly smaller than a jumbo frame - applications are not going to 
change that rapidly to expand the amount of information they exchange that 
would benefit from jumbo frames.  This is true for IPC, file and storage 
applications.  Some performance benefits have been seen using jumbo frames 
for services such as a FTP server, but most people don't have a PMTU that 
will support jumbo frames so the benefit is perhaps confined to a data center.

- Jumbo frames cause head-of-line blocking within a fabric so unless one 
supports multi-pathing / traffic segregation as well as multiple ports on 
each endnode, low-latency / hi-pri traffic QoS will be negatively impacted.

- TCP large send also causes QoS problems as many implementations only work 
at the message level rather than the frame level so differentiated services 
/ bandwidth management are much harder to implement as these become 
competing objectives for the point of serialization, i.e. the link.

While it is true that at higher link rates, the impacts on QoS may be 
minimized but the impacts are not eliminated and one would need to do more 
at the fabric level as well as the NIC implementation level to provide a 
reasonable solution.

As for TOE or RDMA, jumbo isn't really a major benefit.  Work is posted for 
transmission / receipt as single operations and the hardware does the rest 
of the work independent of the application / CPU.  The benefits are 
demonstrated using existing technology such as InfiniBand (numerous 
performance analysis papers are publicly available on Linux based 
implementations) and on some early TCP-based protocol off-load prototypes 
which confirm similar savings as found with InfiniBand.  I think it is fair 
to say that for selected applications, protocol stack off-load of at least 
the main data path will result in reduced CPU consumption, reduced memory 
bus and I/O bus consumption, and improved application performance.  Whether 
this is worthwhile or not is really a judgement call and is somewhat a 
function of whether one is changing only one end of the wire ala a TOE or 
both ends ala using a RDMA wire protocol on top of a protocol off-load 
device.  Embedded or host-based (e.g. a server) will see similar resource 
savings on one hand and will pay a price for those savings on the other 
hand, i.e. dealing with state and resource management in what is generally 
implemented as a non-coherent device.

Mike