Subject: RE: [Tsvwg] Re: [e2e] Are you interested in TOEs and related issues

From: "Leonid Grossman" <[email-masked]>
Date: Thu, 4 Mar 2004 08:38:27 -0800

> -----Original Message-----
> From: Luigi Rizzo [mailto:[email-masked]] 
> Sent: Thursday, March 04, 2004 7:12 AM
> To: H.K. Jerry Chu
> Cc: [email-masked]; [email-masked]; [email-masked]; 
> [email-masked]
> Subject: Re: [Tsvwg] Re: [e2e] Are you interested in TOEs and 
> related issues
> 
> 
> On Thu, Mar 04, 2004 at 11:11:36PM -0800, H.K. Jerry Chu wrote: ...
> > Bursty traffic may be ok for LAN, but is considered evil 
> for WAN. TOE 
> > doesn't suffer this problem (since it is stateful), and may be a 
> > necessary evil until jumbo frames become universal.
> 
> actually the interesting observation that this discussion 
> raises is that the frame size/link MTU should be picked large 
> enough to make the end-to-end pps rate bounded (but not too 
> low) for well behaved flows.
> 
> In 1980 the ethernet MTU allowed some 800pps -- going much 
> higher makes little sense anyways, because even if the end 
> nodes can cope with higher rates, the control loops cannot 
> not react fast enough (you have to factor in propagation delays too).
> 
> On the negative side, however, is that that very large MTUs 
> require a lot of buffering to be preallocated on the receive 
> side, and possibly even at the routers (at least, those 
> without hw-assisted forwarding planes).
> 
> I don't have a good answer but going much higher than 
> 16Kbytes MTUs seems unlikely... and at 10Gig this is still 
> close to 100kpps.

I think we'll be stuck at 9Kbytes for a while because of the CRC
protection (or rather the weakening of such) for 16Kbytes frames, at
least for 'simple' errors.
Leonid

> 	cheers
> 	luigi

_______________________________________________
tsvwg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tsvwg

---

From: "H.K. Jerry Chu" <[email-masked]>
Date: Thu, 4 Mar 2004 23:11:36 -0800 (PST)

><SNIP>
...
>> >The gains are by optimizing data movement and not by
>> >    offloading protocol processing.
>> 
>> LSO is a 'stateless' offload (much like checksum offload) so one doesn't
>> need a TOE to support it; 
>
>It depends. If the kernel does a write based on open window, then yes
>you are right. But if you have full TOE and you pass large chunks of
>data to TOE (does wonder when sendfile/sendfilev is in play) and let
>TOE figure out when to send it, the gains increase. In the later case,
>TOE needs teh TCP state.
>
>> most modern NICs (including our 10GbE Adapter) do LSO. Also, for normal
>> frames LSO has quite dramatic effect at 10GbE rates, but for Jumbo
>> frames performance gain diminishes to ~7%.

One fundamental performance problem high speed Ethernet imposes on
the hosts is from the tiny size of the standard Ethernet frames.
It's almost like throwing ATM cells at the host 10 years ago but
without any hardware adaptation.

LSO and ack coalescing are the adpatation layer for Ethernet cells.
They can help to alleviate this problem but unfortunately they either
induce, or live on traffic bursts. (Just imagine how useful LSO will
be if the ack-every-other-pkt policy provides only 2-MSS worth of
usable window, and how much back-to-back bursts will be caused by
coaliscing many acks.)

Bursty traffic may be ok for LAN, but is considered evil for WAN.
TOE doesn't suffer this problem (since it is stateful), and may be
a necessary evil until jumbo frames become universal.

Jerry

>
>Do you have any insight on why you would see a degradation with Jumbo
>frames? As long as you are stateless, the jumbograms should show
>no degradation. 
>
>Thanks,
>Sunay
>
>
>> 
>> 
>> > 5) TOE is necessary for RDMA, iSCSI etc. for layering 
>> > reasons. I am not
>> >    involved with RDMA so someone who is an expert can 
>> > probably comment on
>> >    this part.
>> > 6) TOE based NIC are already making pretty good headway in 
>> > embedded space.
>> >    The technology is already maturing so why not use it in 
>> > broader market.
>> > 
>> > Note that the above claims are in no particular order of 
>> > importance and made my TOE vendors in general. Of these, I 
>> > personally do agree with 1 and 4 but that iteself doesn't 
>> > mean that TOE will make it in general purpose networking.
>> > 
>> > It would be interesting to see if you and others in the list 
>> > agree or disagree with these claims.
>> > 
>> > Thanks,
>> > Sunay
>> > 
>> > -- 
>> > 
>> > Sunay Tripathi
>> > Senior Staff Engineer,
>> > Solaris Kernel Networking,
>> > Sun MicroSystems Inc.
>> > 
>> > email: [email-masked]		 Phone:	650-786-6007 (W)
>> > 
>> > 
>> > 
>> > 
>> 
>> 
>> Regards, 
>> Leonid Grossman
>> www.s2io.com
>> 
>> 
>-- 
>Sunay Tripathi
>Solaris Kernel Networking,
>Sun MicroSystems Inc.
>
>email: [email-masked]		 Phone:	650-786-6007 (W)
>
>
>

---

From: Luigi Rizzo <[email-masked]>
Date: Thu, 4 Mar 2004 07:12:16 -0800

On Thu, Mar 04, 2004 at 11:11:36PM -0800, H.K. Jerry Chu wrote:
...
> a necessary evil until jumbo frames become universal.

actually the interesting observation that this discussion raises
is that the frame size/link MTU should be picked large enough to make the
end-to-end pps rate bounded (but not too low) for well behaved flows.

In 1980 the ethernet MTU allowed some 800pps -- going much higher makes
little sense anyways, because even if the end nodes can cope with
higher rates, the control loops cannot not react fast enough
(you have to factor in propagation delays too).

On the negative side, however, is that that very large MTUs require
a lot of buffering to be preallocated on the receive side, and
possibly even at the routers (at least, those without hw-assisted
forwarding planes).

I don't have a good answer but going much higher than 16Kbytes MTUs
seems unlikely... and at 10Gig this is still close to 100kpps.

	cheers
	luigi