Subject: Re: 50000 connections and gigabit speeds
From: [email-masked]
To: [email-masked] (Rob Warnock)
Cc: [email-masked]
Date: Fri, 18 Dec 92 8:57:57 CST

> 
> Do you see now?
> 
> 
> -Rob
> 
Yes I do. Thanks for the info, it pointed out a few things that I had not 
thought of.

Kurt

To: [email-masked]
Reply-To: [email-masked] (Rob Warnock)
From: [email-masked] (Rob Warnock)
Subject: Re: 50000 connections and gigabit speeds
Date: Fri, 18 Dec 1992 07:08:14 GMT

[email-masked] writes:
+---------------
| > Even in a LAN, increases in link speed give disappointingly small decreases
| > in latency. Just ask anybody about NFS over Ethernet versus over FDDI...
| > Gigabits are for blasting files, not transactions [unless you have 50,000
| > outstanding transactions or something]....
| 
| By this logic I should be using 2400 baud lines instead of Ethernet or
| FDDI? :-)   No, seriously, am I missing something?
+---------------

I think you may be. Look again at what I said before:

 "Even in a LAN, increases in link speed give disappointingly small decreases
  in latency. Just ask anybody about NFS over Ethernet versus over FDDI..."

Certainly, you should consider higher speeds than 2400 baud(!), but in most
LAN environments latency is a more potent inhibitor than slow link speed.
Increasing your link speed only decreases the time it takes to transmit a
single packet. It *doesn't* decrease the time it takes for the far end to
parse, act on, and respond to that packet, and the near end to parse and
act on the reply. (See below for the rest.)

+---------------
| Can you only get gigabit speeds out of WANs? Why can't we build a gigabit
| LAN if we can build a gigabit WAN?
+---------------

Of course gigabit LANs are simpler/easier/cheaper to build than gigabit WANs.
That's not the point. The point is that even *LANs* are latency-limited, at
gigabit speeds.

+---------------
| On another note, I think that we can drop the discussion of 50,000 connections
| in a transaction environment here. TCP large windows aren't going to be 
| helpful in that situation anyway. We are getting sidetracked on an issue that
| doesn't really belong here.
+---------------

But it *does* belong here! You're missing the fact that increased bandwidth
*only* increases performance if the amount of data "in flight" per round-trip-
time can increase, which can only happen one of three ways:

1. For a "windowed/streaming" protocol (TCP & others like it), if you
   increase the "window size", so you keep the faster pipe "full" while
   waiting for the ACKs from the remote end that *still* are going to
   take a while to return (since increased bandwidth doesn't help protocol
   handling times or transmission times for small things like ACKs).

   The whole point of TCP large windows is to "keep the pipe full" by allowing
   more than the delay-bandwidth product of data to be outstanding at once.
   Even on a local, fairly small FDDI ring, we have seen situations where
   TCP large windows were required to achieve full link rate. At gigabit
   speeds, it's a "must".

2. If you *are* stuck with a stop&wait request/response protocol (e.g., NFS,
   most RPCs), you can try increasing the *size* of each request so that at
   the faster link speed each request still takes about the same amount of
   time to send or receive.

   But there's a limit to how big you can make the individual RPC requests/
   responses. And if the practical request/response size at the faster bit
   rate is much less than the round-trip latency (which can easily happen
   for, say, NFS on a gigabit LAN), then the throughput is dominated by the
   round-trip latency, and is fairly independent of the actual bitrate of
   the LAN (provided it's above some minimum speed, of course).
   
   For NFS, for example, experience so far puts the "knee" of the curve
   somewhere between Ethernet and FDDI. I know of no-one who has yet gotten
   more than a small fraction of an FDDI's worth of NFS traffic on a single
   client/server connection.

3. If you are stuck with a stop&wait request/response protocol with *really*
   small transactions compared to the bandwidth*delay product (e.g., typical
   X Windows packets even on Ethernet, or NFS over FDDI or faster), then the
   only way to increase total throughput is to do a *lot* of these small
   things in parallel, which is where the notion of "50,000 connections
   in a transaction environment" comes from.

   The latency of each individual connection is still long, and each individual
   request/response client/server connection's throughput is still slow, but
   at least you can make some use of your hundreds of megabits or your gigabits
   by having *lots* of simultaneously outstanding transactions.

When the delay-bandwidth product is *really* large (megabytes), then trying
to get link-speed performance out of a request/response transaction protocol
is ludicrous (for a single client/server connection). You might as well send
the entire database to the client with a windowed transport protocol (e.g.,
TCP), have him do a few transactions locally, and send the entire database back
to the server (also with TCP or equiv). [This has been seriously considered
in some environments!]

Due to latency being essentially constant, there is always *some* speed
above which further increases in bitrate do not substantially increase
request/response transaction performance, but may (no promises!) increase
a windowed/streaming protocol's performance (if the window is large enough).

Do you see now?

-Rob

-----
Rob Warnock, MS-9U/510		[email-masked]
Silicon Graphics, Inc.		(415)390-1673
2011 N. Shoreline Blvd.
Mountain View, CA  94043

To: [email-masked]
Reply-To: [email-masked] (Rob Warnock)
From: [email-masked] (Rob Warnock)
Subject: Re: 50000 connections and gigabit speeds
Date: Fri, 18 Dec 1992 06:59:08 GMT

[email-masked] writes:
+---------------
| > Even in a LAN, increases in link speed give disappointingly small decreases
| > in latency. Just ask anybody about NFS over Ethernet versus over FDDI...
| > 
| > Gigabits are for blasting files, not transactions [unless you have 50,000
| > outstanding transactions or something]....
| 
| By this logic I should be using 2400 baud lines instead of Ethernet or
| FDDI? :-)   No, seriously, am I missing something?
+---------------

I think you may be. Look again at what I said before:

 "Even in a LAN, increases in link speed give disappointingly small decreases
  in latency. Just ask anybody about NFS over Ethernet versus over FDDI..."

Certainly, you should consider higher speeds than 2400 baud(!), but in most
LAN environments latency is a more potent inhibitor than slow link speed.
Increasing your link speed only decreases the time it takes to transmit a
single packet. It *doesn't* decrease the time it takes for the far end to
parse, act on, and respond to that packet, and the near end to parse and
act on the reply. (See below for the rest.)

+---------------
| Can you only get gigabit speeds out of WANs? Why can't we build a gigabit
| LAN if we can build a gigabit WAN?
+---------------

Of course gigabit LANs are simpler/easier/cheaper to build than gigabit WANs.
That's not the point. The point is that even *LANs* are latency-limited, at
gigabit speeds.

+---------------
| On another note, I think that we can drop the discussion of 50,000 connections
| in a transaction environment here. TCP large windows aren't going to be 
| helpful in that situation anyway. We are getting sidetracked on an issue that
| doesn't really belong here.
+---------------

But it *does* belong here! You're missing the fact that increased bandwidth
*only* increases performance if the amount of data "in flight" per round-trip-
time can increase, which can only happen one of three ways:

1. For a "windowed/streaming" protocol (TCP & others like it), if you
   increase the "window size", so you keep the faster pipe "full" while
   waiting for the ACKs from the remote end that *still* are going to
   take a while to return (since increased bandwidth doesn't help protocol
   handling times or transmission times for small things like ACKs).

   The whole point of TCP large windows is to "keep the pipe full" by allowing
   more than the delay-bandwidth product of data to be outstanding at once.
   Even on a local, fairly small FDDI ring, we have seen situations where
   TCP large windows were required to achieve full link rate. At gigabit
   speeds, it's a "must".

2. If you *are* stuck with a stop&wait request/response protocol (e.g., NFS,
   most RPCs), you can try increasing the *size* of each request so that at
   the faster link speed each request still takes about the same amount of
   time to send or receive.

   But there's a limit to how big you can make the individual RPC requests/
   responses. And if the practical request/response size at the faster bit
   rate is much less than the round-trip latency (which can easily happen
   for, say, NFS on a gigabit LAN), then the throughput is dominated by the
   round-trip latency, and is fairly independent of the actual bitrate of
   the LAN (provided it's above some minimum speed, of course).
   
   For NFS, for example, experience so far puts the "knee" of the curve
   somewhere between Ethernet and FDDI. I know of no-one who has yet gotten
   more than a small fraction of an FDDI's worth of NFS traffic on a single
   client/server connection.

3. If you are stuck with a stop&wait request/response protocol with *really*
   small transactions compared to the bandwidth*delay product (e.g., typical
   X Windows packets even on Ethernet, or NFS over FDDI or faster), then the
   only way to increase total throughput is to do a *lot* of these small
   things in parallel, which is where the notion of "50,000 connections
   in a transaction environment" comes from.

   The latency of each individual connection is still long, and each individual
   request/response client/server connection's throughput is still slow, but
   at least you can make some use of your hundreds of megabits or your gigabits
   by aggregating lots of sessions.

When the delay-bandwidth product is *really* large (megabytes), then trying
to get link-speed performance out of a request/response transaction protocol
is ludicrous (for a single client/server connection). You might as well send
the entire database to the client with a windowed transport protocol (e.g.,
TCP), have him do a few transactions locally, and send the entire database back
to the server (also with TCP or equiv). [This has been seriously considered
in some environments!]

Due to latency being essentially constant, there is always *some* speed
above which further increases in bitrate do not substantially increase
request/response transaction performance, but may (no promises!) increase
a windowed/streaming protocol's performance (if the window is large enough).

Do you see now?

-Rob

-----
Rob Warnock, MS-9U/510		[email-masked]
Silicon Graphics, Inc.		(415)390-1673
2011 N. Shoreline Blvd.
Mountain View, CA  94043

Date: Thu, 17 Dec 92 23:02:14 -0800
From: [email-masked] (Rob Warnock)
To: [email-masked]
Subject: Re: 50000 connections and gigabit speeds
Cc: [email-masked]

[email-masked] writes:
+---------------
| > Even in a LAN, increases in link speed give disappointingly small decreases
| > in latency. Just ask anybody about NFS over Ethernet versus over FDDI...
| > Gigabits are for blasting files, not transactions [unless you have 50,000
| > outstanding transactions or something]....
| 
| By this logic I should be using 2400 baud lines instead of Ethernet or
| FDDI? :-)   No, seriously, am I missing something?
+---------------

I think you may be. Look again at what I said before:

 "Even in a LAN, increases in link speed give disappointingly small decreases
  in latency. Just ask anybody about NFS over Ethernet versus over FDDI..."

Certainly, you should consider higher speeds than 2400 baud(!), but in most
LAN environments latency is a more potent inhibitor than slow link speed.
Increasing your link speed only decreases the time it takes to transmit a
single packet. It *doesn't* decrease the time it takes for the far end to
parse, act on, and respond to that packet, and the near end to parse and
act on the reply. (See below for the rest.)

+---------------
| Can you only get gigabit speeds out of WANs? Why can't we build a gigabit
| LAN if we can build a gigabit WAN?
+---------------

Of course gigabit LANs are simpler/easier/cheaper to build than gigabit WANs.
That's not the point. The point is that even *LANs* are latency-limited, at
gigabit speeds.

+---------------
| On another note, I think that we can drop the discussion of 50,000 connections
| in a transaction environment here. TCP large windows aren't going to be 
| helpful in that situation anyway. We are getting sidetracked on an issue that
| doesn't really belong here.
+---------------

But it *does* belong here! You're missing the fact that increased bandwidth
*only* increases performance if the amount of data "in flight" per round-trip-
time can increase, which can only happen one of three ways:

1. For a "windowed/streaming" protocol (TCP & others like it), if you
   increase the "window size", so you keep the faster pipe "full" while
   waiting for the ACKs from the remote end that *still* are going to
   take a while to return (since increased bandwidth doesn't help protocol
   handling times or transmission times for small things like ACKs).

   The whole point of TCP large windows is to "keep the pipe full" by allowing
   more than the delay-bandwidth product of data to be outstanding at once.
   Even on a local, fairly small FDDI ring, we have seen situations where
   TCP large windows were required to achieve full link rate. At gigabit
   speeds, it's a "must".

2. If you *are* stuck with a stop&wait request/response protocol (e.g., NFS,
   most RPCs), you can try increasing the *size* of each request so that at
   the faster link speed each request still takes about the same amount of
   time to send or receive.

   But there's a limit to how big you can make the individual RPC requests/
   responses. And if the practical request/response size at the faster bit
   rate is much less than the round-trip latency (which can easily happen
   for, say, NFS on a gigabit LAN), then the throughput is dominated by the
   round-trip latency, and is fairly independent of the actual bitrate of
   the LAN (provided it's above some minimum speed, of course).
   
   For NFS, for example, experience so far puts the "knee" of the curve
   somewhere between Ethernet and FDDI. I know of no-one who has yet gotten
   more than a small fraction of an FDDI's worth of NFS traffic on a single
   client/server connection.

3. If you are stuck with a stop&wait request/response protocol with *really*
   small transactions compared to the bandwidth*delay product (e.g., typical
   X Windows packets even on Ethernet, or NFS over FDDI or faster), then the
   only way to increase total throughput is to do a *lot* of these small
   things in parallel, which is where the notion of "50,000 connections
   in a transaction environment" comes from.

   The latency of each individual connection is still long, and each individual
   request/response client/server connection's throughput is still slow, but
   at least you can make some use of your hundreds of megabits or your gigabits
   by having *lots* of simultaneously outstanding transactions.

When the delay-bandwidth product is *really* large (megabytes), then trying
to get link-speed performance out of a request/response transaction protocol
is ludicrous (for a single client/server connection). You might as well send
the entire database to the client with a windowed transport protocol (e.g.,
TCP), have him do a few transactions locally, and send the entire database back
to the server (also with TCP or equiv). [This has been seriously considered
in some environments!]

Due to latency being essentially constant, there is always *some* speed
above which further increases in bitrate do not substantially increase
request/response transaction performance, but may (no promises!) increase
a windowed/streaming protocol's performance (if the window is large enough).

Do you see now?

-Rob

-----
Rob Warnock, MS-9U/510		[email-masked]
Silicon Graphics, Inc.		(415)390-1673
2011 N. Shoreline Blvd.
Mountain View, CA  94043

Date: Thu, 10 Dec 1992 14:24:10 -0800
From: [email-masked] (Bob Braden)
To: [email-masked], [email-masked]
Subject: Re: 1323 implementations

> 
> 	Name: _____________Bob Braden, Liming Wei_____________
> 
> 	Organization: _____Information Sciences Institute (ISI)
> 
> 	Email/Phone: [email-masked], [email-masked]
> 
> 	[   ] This information should be kept confidential
> 
> 	I have implemented:
> 		[ X ]	TCP Window Scale Option
> 		[ X ]	TCP Timestamps Option
> 		[ X ]	Am using Timestamps for RTT measurements.
> 		[ X ]	Am using Timestamps PAWS.
> 
> 	OS name/version: __Sun OS 4.1____________
> 	The code is:
> 		[ X ]	Experimental/research/prototype.
> 		[   ]	Will be/is a Product.
> 		[   ]	Not available for distribution.
> 		[ X ]	Will be/is available AS IS.
> 		[   ]	Will be/is available for purchase.
> 		[   ]	There is support for the code.
> 
> 	Availablity Information (date available, OS version,
> 	how to get the code, who can get the code):
> 
> 	___Available Jan. 93, or when someone asks, whichever is
> 
> 	___later.__________________________________________________
> 
> 	[    ]	There are known deficiencies in the implementation.
> 
> 		Please explain. ____________________________________
> 
> 		____________________________________________________
> 
> 
> 	[    ]  I have tested my code against other implementations.
> 
> 		Please list: _______________________________________
> 
> 		____________________________________________________
> 

Date: Tue, 8 Dec 1992 15:07:40 -0600
From: [email-masked] (David A. Borman)
To: [email-masked]
Subject: 1323 implementations

Hi.

It has now been 6 months since RFC 1323 (TCP Extensions for High
Performance) was published, so it how now met the minimum time
requirement before being allowed to be advanced from a Proposed
Standard to a Draft Standard.

Before it can be elevated to Draft, there have to be at least two
independent and interoperable implementations.

So, I am taking an implementation survey.  If you check the box
to keep the information confidential, then the only information
that I will make public is that there is an implementation, and
which parts of the RFC it supports.  Names/Organization/OS will
not be mentioned.

If you do not check the box, then I will include that information
in my summary of the results back to the mailing list.

I will not include the names from the interoperability testing
information unless I get conformation from the other implementor
that it is ok to release that information, so don't worry about
citing other implementations that you have tested against.

I've attached a sample reply at the end...

			-David Borman, [email-masked]

-------- Cut here and return to [email-masked] -----------

	Name: _____________________________________________

	Organization: _____________________________________

	Email/Phone: ______________________________________

	[   ] This information should be kept confidential

	I have implemented:
		[   ]	TCP Window Scale Option
		[   ]	TCP Timestamps Option
		[   ]	Am using Timestamps for RTT measurements.
		[   ]	Am using Timestamps PAWS.

	OS name/version: ____________________________
	The code is:
		[   ]	Experimental/research/prototype.
		[   ]	Will be/is a Product.
		[   ]	Not available for distribution.
		[   ]	Will be/is available AS IS.
		[   ]	Will be/is available for purchase.
		[   ]	There is support for the code.

	Availablity Information (date available, OS version,
	how to get the code, who can get the code):

	____________________________________________________________

	____________________________________________________________

	[    ]	There are known deficiencies in the implementation.

		Please explain. ____________________________________

		____________________________________________________

	[    ]  I have tested my code against other implementations.

		Please list: _______________________________________

		____________________________________________________

---------- Cut here for Sample form --------------

	Name: David Borman

	Organization: Cray Research, Inc.

	Email/Phone: [email-masked] (612)683-5571

	[   ] This information should be kept confidential

	I have implemented:
		[ X ]	TCP Window Scale Option
		[ X ]	TCP Timestamps Option
		[ X ]	Am using Timestamps for RTT measurements.
		[ X ]	Am using Timestamps PAWS.

	OS name/version: UNICOS 6.1 and later
	The code is:
		[   ]	Experimental/research/prototype.
		[ X ]	Will be/is a Product.
		[   ]	Not available for distribution.
		[   ]	Will be/is available AS IS.
		[   ]	Will be/is available for purchase.
		[ X ]	There is support for the code.

	Availablity Information (date available, OS version,
	how to get the code, who can get the code):

		The code is part of the regular UNICOS distributions,
		and is available today with Unicos 6.1, 7.0 and 7.C

	[ X  ]	There are known deficiencies in the implementation.

		The code must be enabled via an explicit setsockopt()
		call by the application.  Just making the kernel
		buffers large will not do it.

		The code in 6.1 uses the Echo and Echo Reply options
		instead of the Timestamps option, and does not support
		PAWS.  There are also mods available for 6.0 and 5.1,
		but these are not supported and still have bugs that
		are fixed in the 6.1 code.  The 7.0 code supports
		Window Scale, Timestamps, RTT calculations and PAWS,
		and also has support for the Echo and Echo Reply
		options for backwards compatability.

	[ X  ]  I have tested my code against other implementations.

		Please list:
			Van Jacobson's code
			At least 3 other partial/complete
			implementations.

Date: Mon, 7 Dec 92 11:01:14 MST
From: [email-masked] (C. Douglas Brown)
To: [email-masked]
Subject: Latency and Gigabit speeds

> By this logic I should be using 2400 baud lines instead of Ethernet or FDDI? :-)
> No, seriously, am I missing something? Can you only get gigabit speeds out of
> WANs? Why can't we build a gigabit LAN if we can build a gigabit WAN?
> 
> On another note, I think that we can drop the discussion of 50,000 connections
> in a transaction environment here. TCP large windows aren't going to be 
> helpful in that situation anyway. We are getting sidetracked on an issue that
> doesn't really belong here.
> 
> Kurt 

I don't think anyone is trying to say you can't build gigabit LANs, but Rob
has a valid point that simply increasing the speed of a network (LAN or WAN)
doesn't necessarily decrease its latency.  In fact, most of the delays in a
network remain pretty constant when you increase the bit rate, except for 
packet transmission time.  This means that the TCP large windows will even be
needed for gigabit LANs, as well as WANs, because a sender might fill the 
standard 64K window before an acknowledgement could be received .

Doug Brown
Sandia National Labs

Subject: Re: 50000 connections and gigabit speeds
To: [email-masked] (Rob Warnock)
Date: Fri, 4 Dec 92 10:06:58 CST
Cc: [email-masked]
From: [email-masked]

> 
> [email-masked] says:
> +---------------
> | > > It is highly unlikely that one would want to run 
> | > > 50,000 tcp connections at gigabit speeds (you would need a 
> | > > lot of buffer space).                                    
> |
> | Why not? The gigabit speed would just decrease the network latency time, 
> | wouldn't it? Then you could get the same transaction rate on the host with
> | fewer networks. The buffer space needed would not go up. I am interested in
> | a transaction environment only, not doing 50,000 file transfers at the same
> | time.
> +---------------
> 
> Then you are going to be mightily disappointed in gigabit networks. Except
> for some few cases of mongo-gram transactions in low-diameter LANs, you will
> not find gigabit networks decreasing latency much. Consider: An 800 mile T3
> link (through two routers) is about 25 ms [as reported by [email-masked],
> now [email-masked]]. That latency will not lessen (and it probably will even
> even go up a bit) if you run over an OC-12 ATM link (622 Mb/s). Unless each
> separate "transaction" is *huge* [which creates error & flow control problems],
> an increase in link speed beyond double-digit megabits/sec will not buy you
> much of anything.
> 
> Even in a LAN, increases in link speed give disappointingly small decreases
> in latency. Just ask anybody about NFS over Ethernet versus over FDDI...
> 
> Gigabits are for blasting files, not transactions [unless you have 50,000
> outstanding transactions or something]....
> 
> 
> -Rob
> 
By this logic I should be using 2400 baud lines instead of Ethernet or FDDI? :-)
No, seriously, am I missing something? Can you only get gigabit speeds out of
WANs? Why can't we build a gigabit LAN if we can build a gigabit WAN?

On another note, I think that we can drop the discussion of 50,000 connections
in a transaction environment here. TCP large windows aren't going to be 
helpful in that situation anyway. We are getting sidetracked on an issue that
doesn't really belong here.

Kurt 
> -----
> Rob Warnock, MS-9U/510		[email-masked]
> Silicon Graphics, Inc.		(415)390-1673
> 2011 N. Shoreline Blvd.
> Mountain View, CA  94043
> 
> 
> 
> 
>