Subject: [Tsvwg] Effects of port randomization with TCP TIME-WAIT state.

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Tue, 8 Jul 2008 23:50:28 -0700

Hi,
    We would like to share some of the field experience on doing port
randomization. We had implemented source port randomization a couple of
years back. Before this the port allocation was sequential and the
moment this was changed to random caused issues w.r.t some applications
which made TCP connections at a high rate. The latest copy of the draft
http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-port-randomization-
01.txt
acknowledges the TIME-WAIT issue briefly (port reuse cycle), but doesn't
describe the issue in detail or offer possible solutions. We had
implemented some of the schemes to circumvent the issues caused by port
randomization on TIME-WAIT state in our TCP implementation.  We have
submitted a draft which summarizes the problem and suggests some
solutions to the it:

http://www.ietf.org/internet-drafts/draft-ananth-tsvwg-timewait-00.txt

We have implemented some of these solutions to circumvent the issues
caused by port randomization on TIME-WAIT state in our TCP stack and it
is been running in the internet for more than an year. We feel this
issue needs to be documented, the main motivation of writing this draft
comes from the experiences which we observed on some customer
deployments.

It needs to be stated that, there are other issues w.r.t TIME-WAIT state
like the one described in Ted Faber et al., paper on timewait loading on
busy servers. The current draft restricts the focus towards the effects
of port randomization on TIME-WAIT state.

Comments welcome.

Thanks,
Patrick and Anantha

---

From: Fernando Gont <[email-masked]>
Date: Wed, 09 Jul 2008 18:35:45 -0300

Hello, Anantha,

>describe the issue in detail or offer possible solutions.

This issue is documented to some extent in Section 3.1 of the draft. 
The issues you point out in Section 2 of your draft (ignoring 
incoming SYNS and TIME-WAIT assassination) were mentioned in the 
discussions we had on this mailing-list. But to the extent that was 
possible, I tried to add more text to the draft only when necessary.

Some comments about your draft:

* In practice, the 2*MSL value in some systems is much smaller than 4 
minutes. e.g.,, at least at some point tee was an idea to make 
FreeBSD's TIME-WAIT state last for only 30 seconds or so.

* The solution in Section 3.2.1 of your draft seems to miss an 
important issue here: only the resulting *five*-tuple needs to be 
unique. At any point in time, the same port number could be used for 
more than one connection. If the solution in sectio 3.2.1 is 
implemented, then the maximum connection rate would be something like 
65K/2*MSL = 273 connections/second. Another issue here is that if you 
implement simple port randomization (described in Section 3.3.1 of 
*our* draft), performance may be degraded (you may have to try quite 
a few ports until you find an available one). And if you implement 
the algorithm in Section 3.3.2 of *our* draft, things get worse: the 
more ports there are in use, the harder it gets to find an "available 
port". Also, if you try random ports one after another (Section 3.3.2 
of our draft), there no upper bound on the number of times you may 
have to retry to find an available port.

* Another interesting point to note about of the algorithms you 
describe  in your draft is that if all port numbers are supposed to 
be "in use", the port selection function will possibly return an 
error. But probably the most sensible thing to do in this case is 
that, for those port numbers that are in use by a remote system (that 
is, where there's a socket in the TIME-WAIT state at the remote 
endpoint), one could reuse them, as must likely the sequence-numbers 
BSD-heuristics will allow the port number to be reused, with no harm.

* Virtually all implementations have implemented an incremental 
port-selection algorithm. From the point-of-view of interoperability, 
it has worked great. That's the rationale for the algorithm in 
Section 3.3.3 of our draft (hash-based for selection algorithm). It 
selects random port numbers without the need for timers or anything.

* The algorithm in Section 3.3.4 of our draft takes that further, 
realizing that there's no need for a single system-wide counter, as a 
system-wide counter skips some ports, unnecessarily.

I doubt that, by the time you implement algorithms 3.3.3 or 3.3.4 
from our draft you'd find the need for timers or the like. The 
problems you describe arise from the use of the simple port 
randomization algorithm.

FWIW, algorithm 3.3.3 from our draft has been implemented in Linux 
since 2004 or so.

That said:

* I did talk with some developers about Method 1 (section 3.2.1) of 
your draft. But it was generally thought as too complex compared with 
the algorithms described in Section 3.3.3 and Section 3.3.4 of our draft.

* Now that you raised this issue, I can include more documentation 
about this problem if the wg agrees. I would copy&paste some text 
from some other TCP stuff I worked on (yes, the PDF).

* Also, if the wg wants, and you have no problem with it, I could 
document some of the alternative mechanisms. Not the "potential" 
ones, but just the ones that have been deployed.

Thanks!

Kind regards,

--
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Wed, 9 Jul 2008 19:06:43 -0700

Fernando,
   Pl find inline comments : 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Fernando Gont
> Sent: Wednesday, July 09, 2008 2:36 PM
> To: Anantha Ramaiah (ananth); [email-masked]
> Cc: [email-masked]; James Polk (jmpolk)
> Subject: Re: [Tsvwg] Effects of port randomization with TCP 
> TIME-WAIT state.
> 
> Hello, Anantha,
> 
> >http://www.ietf.org/internet-drafts/draft-ietf-tsvwg-port-ran
> domization
> >-
> >01.txt
> >acknowledges the TIME-WAIT issue briefly (port reuse cycle), but 
> >doesn't describe the issue in detail or offer possible solutions.
> 
> incoming SYNS and TIME-WAIT assassination) were mentioned in 
> the discussions we had on this mailing-list. But to the 
> extent that was possible, I tried to add more text to the 
> draft only when necessary.
> 
> * In practice, the 2*MSL value in some systems is much 
> smaller than 4 minutes. e.g.,, at least at some point tee was 
> an idea to make FreeBSD's TIME-WAIT state last for only 30 
> seconds or so.

Right, but this is something which the client doesn't know [ unless you
have a TCP option to exchange the TIME-WAIT value] I have seen some
versions of Windows also default the TIME-WAIT value to 30 secs. 

> 
> * The solution in Section 3.2.1 of your draft seems to miss 
> an important issue here: only the resulting *five*-tuple 
> needs to be unique. At any point in time, the same port 
> number could be used for more than one connection. If the

Oh, I thought this is implied in the document. ie., the typical case of
clients trying to establish connections with the server at a high rate,
which implies the source port is the only thing which varies and the
rest (src addr, dest addr and dest port) is the same. Do you think this
needs to be mentioned explicitly in the document ?
 
> solution in sectio 3.2.1 is implemented, then the maximum 
> connection rate would be something like 65K/2*MSL = 273 
> connections/second. Another issue here is that if you 

Well, this may be the worst case, but again it depends on many factors
like for example how many connections are actively versus passively
closed etc.,

> *our* draft), performance may be degraded (you may have to 
> try quite a few ports until you find an available one). And 
> if you implement the algorithm in Section 3.3.2 of *our* 
> draft, things get worse: the more ports there are in use, the 
> harder it gets to find an "available port". Also, if you try 
> random ports one after another (Section 3.3.2 of our draft), 
> there no upper bound on the number of times you may have to 
> retry to find an available port.

Ok.

> 
> * Another interesting point to note about of the algorithms 
> you describe  in your draft is that if all port numbers are 
> supposed to be "in use", the port selection function will 
> possibly return an error. But probably the most sensible 
> thing to do in this case is that, for those port numbers that 
> are in use by a remote system (that is, where there's a 
> socket in the TIME-WAIT state at the remote endpoint), one 

Again you are assuming all connections are passively closed here.

> could reuse them, as must likely the sequence-numbers 

This is another dimension to the problem, a) you are relying on the seq#
which is again chosen randomly to fall in the "favourable window" AND
you are assuming that TCP stack implementation allows re-incarnation.

> 
> * Virtually all implementations have implemented an 
> incremental port-selection algorithm. From the point-of-view 
> of interoperability, it has worked great. That's the 
> rationale for the algorithm in Section 3.3.3 of our draft 
> (hash-based for selection algorithm). It selects random port 
> numbers without the need for timers or anything.
> 
> * The algorithm in Section 3.3.4 of our draft takes that 
> further, realizing that there's no need for a single 
> system-wide counter, as a system-wide counter skips some 
> ports, unnecessarily.
> 
> I doubt that, by the time you implement algorithms 3.3.3 or 
> 3.3.4 from our draft you'd find the need for timers or the 
> like. The problems you describe arise from the use of the 
> simple port randomization algorithm.

Hmm.. For a stateless random number generator which doesn't take into
account the TIME-WAIT property would suffer the birthday paradox issue.
Also if I understood your 3.3.3 correctly, it sacrifices some randomess
since you mention the "function F() provides a per-connection fixed
offset within the global ephemeral port range." FWIW, the method
mentioned in 3.2.2 in our draft doesn't require an "active" running
timer.

> 
> FWIW, algorithm 3.3.3 from our draft has been implemented in 
> Linux since 2004 or so.
> 
> That said:
> 
> * I did talk with some developers about Method 1 (section 
> 3.2.1) of your draft. But it was generally thought as too 
> complex compared with the algorithms described in Section 
> 3.3.3 and Section 3.3.4 of our draft.

Well the tradeoff is complexity versus the degree of randomness sought.
That said, as far as the implementation goes, it is question of memory
versus performance, for example the lookup in the 2MSL array can be made
more efficient at the expense of memory.

That said, I agree that different systems might have different
requirements w.r.t the degree of randomness sought.

> 
> * Now that you raised this issue, I can include more 
> documentation about this problem if the wg agrees. I would 
> copy&paste some text from some other TCP stuff I worked on 
> (yes, the PDF).
> 
> * Also, if the wg wants, and you have no problem with it, I 
> could document some of the alternative mechanisms. Not the 
> "potential" 

I have no issues here.. Like we mentioned earlier, the method outlined
in our draft is proven to work well and also provide sufficient degree
of randomness. I am not denying the fact that there cannot be more
efficient methods possible, for example we considered some keeping the
2MSL list sorted in descending order i.e., the longlived ones would
picked first if we run out ports like you mention above, but we never
implemented those since in practice what we have proved to be working
fine. 

Thanks for your feedback.

-Anantha
> 
> Thanks!
> 
> Kind regards,
> 
> --
> Fernando Gont
> e-mail: [email-masked] || [email-masked] PGP 
> Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> 
> 
> 
> 
>

---

From: Fernando Gont <[email-masked]>
Date: Thu, 10 Jul 2008 02:51:19 -0300

>versions of Windows also default the TIME-WAIT value to 30 secs.

Agreed. My point was that assuming a TIME-WAIT state of 4 minutes 
might tie port numbers for un unnecessarily long time.

>needs to be mentioned explicitly in the document ?

Yes. Particularly when many implementations require the source port, 
rather than the five-tuple, to be unique.

>like for example how many connections are actively versus passively

Yes. However, if the client gets into the TIME-WAIT state (i.e., 
performs the active close), the situation is even better: you can 
reuse the port number. Just be careful with the ISN you pick.

>Again you are assuming all connections are passively closed here.

Yes, that's the worst-case scenario. The port-number collisions that 
are harder to deal with are those for that result from a remote peers 
having a socket with the same five-tuple: you simply don't know about them.

Now, even if you knew, once you have run out of port numbers, the 
most sensible thing to do is to reuse the port numbers, carefully 
picking the ISN for the new incarnation of the connection. (no matter 
whether the socket in the TIME-WAIT state is local or remote).

>which is again chosen randomly to fall in the "favourable window"

If you don't do RFC1948, you simply run the risk that if your 
connection rate is higher that ephemeral_ports/TIME-WAIT state, your 
connections will start failing. As a matter of fact, IIRC there was a 
presentation by Mike Silbersack at EuroBSDCon on the use of network 
randomness which, IIRC, mentioned this.

FWIW, it is usually assumed that the ISN generation function must be 
linear to avoid reliability problems (processing old segments as 
new). This functionality is actually provided by the TIME-WAIT state 
and the "quiet time" concept, rather than by the ISN-generation 
function. So my point is simply that if you don't do RFC1948, the BSD 
heuristics for reusing port numbers will break.

>AND you are assuming that TCP stack implementation allows re-incarnation.

This is pretty common.

>account the TIME-WAIT property would suffer the birthday paradox issue.

Agreed. And even if it's not stateless (e.g., a PRNG that first uses 
all the numbers in the sequence before reusing a value), you still 
have the problem of reusing a port number too quickly: particularly 
at the border of any two sequences of values.

>offset within the global ephemeral port range."

Simply put, section 3.3.3 is RFC 1948 applied to port numbers. It's 
true that it sacrifices randdomness. Actually, the randomness is in 
the per-destination offset. The generation function is something like 
y = x + b , where "b" is a number that depends on the destination 
(from a secure hash function) and x gets incremented for every connection.

The offset provides the randomness. And the fact that the 
port-generation function is linear gets you the same interoperability 
properties than the BSD port selection function.

>FWIW, the method

Agreed.

>requirements w.r.t the degree of randomness sought.

Agreed.

>implemented those since in practice what we have proved to be working

Great. I plan to work on a revision of the port randomization draft 
anytime soon (there's some pending feedback I got from bugtraq and 
full-disclosure, too), and will come back to you on this issue when I 
start working on that revision.

Bottom-line: I'm happy to have this discussion. It will certainly 
lead to improvements that would mitigate vulnerabilities such as, 
e.g., the recent vulnerability in BIND and others.... which simply 
has to do with not doing port randomization (well, actually worse: 
using a fixed port number for successive DNS requests).

Thanks!

Kind regards,

---

From: "Patrick Tate (ptate)" <[email-masked]>
Date: Thu, 10 Jul 2008 07:07:40 -0400

Sorry, late in the game coming into this discussion ...

I generally agree with the points you both have made below.  
See below for one point ... 

> From: Fernando Gont [mailto:[email-masked]] 
> Sent: Thursday, July 10, 2008 1:51 AM
> Cc: [email-masked]; James Polk (jmpolk); 
> Patrick Tate (ptate)
> smaller than 
> > > 4 minutes. e.g.,, at least at some point tee was an idea to make 
> Agreed. My point was that assuming a TIME-WAIT state of 4 
> minutes might tie port numbers for un unnecessarily long time.
> 
> 
> 

Just want to point out that even algorithms such as 3.3.3 and 3.3.4 
can produce ports that will result in a TIME-WAIT collision if you have 
connections coming up and going down using a similar 5-tuple, but the 
connections are long-lived.  Once the long-lived connections start to go

down and new ephemeral ports are created using one of these approaches, 
next_ephemeral could well be around the value used for these
connections.  
Realizing the issue this way also occurs more readily (as it will with 
any algorithm) if the implementation for some reason cannot take 
advantage of the recommended larger ephemeral port range.

I guess my point here is that hitting this issue depends on several 
factors including connection rate, but also "lifetime" of connection, 
randomness of the ephemeral port algorithm, and range of ephemeral ports

that the client host/application can effectively use.

Our approach and hopefully the end result of our further discussion on 
this would be to find a way that is mostly full-proof for finding a port
that should not result in the TIME-WAIT collision.

I like what you have to say in you point above about how to find a port 
to reuse from the set if you have run out of ports by carefully choosing

the ISN (or perhaps finding the oldest of such ports as Anantha suggests

AND carefully choosing the ISN).  I think if we can find a solution that

attempts to be both mostly full-proof in finding unique ports, and
capable 
of reusing ports smartly when all are in use, this could really be
offered 
as an alternative approach for implementations that need it.

Thanks,
-Patrick

---

From: Fernando Gont <[email-masked]>
Date: Thu, 10 Jul 2008 12:37:57 -0300

>next_ephemeral could well be around the value used for these

Yes, but this is also the case for the traditional BSD algorithm, and 
that is the algorithm that has been in use in most stacks for at 
least the last 20 years. At it has worked fine even when using a very 
small ephemeral port range.

>advantage of the recommended larger ephemeral port range.

I'd sum-up this issue this way:

* Choosing port numbers incrementally is a cheap and effective way to 
reduce the port number reuse frequency. This has been what virtually 
all implementations have been doing for the last 20 years or so.

* While it's great from the pov of interoperability, it's not from 
the pov of security/obfuscation.

* No matter how you pick port numbers (whether with one of the 
algorithms in our draft, or with one of the methods in your draft), 
at the point in which your connection rate is something like 
ephemeral_port_range/time_wait_state , you may run into trouble. 
That's why you should implement the BSD sequence-number heuristics.

* If you implement some mechanism that uses timers (as in any of the 
methods you describe in your draft), you will still face the problem 
I mentioned in the previous bullet when the connection rate 
approaches ephemeral_port_range/time_wait_state. Things *might* be 
better for the scenario in which there are some long lived 
connections and the connection-rate is high enough that you cycle 
through the whole port number space and have to reuse some port 
numbers. However, the timers in your draft are per-port (and not 
per-five-tuple), and thus that itself might greatly increase the port 
reuse frequency in those scenarios in which the connection requests 
are sent to more than one peer (because it does not take advantage of 
the fact that only the five-tuple, rather than the source port, has 
to be unique).

>randomness of the ephemeral port algorithm, and range of ephemeral ports

I agree.

>that should not result in the TIME-WAIT collision.

Clearly, collisions should be avoided. But no matter what you do, if 
the connection-rate is high enough, collisions will take place. And 
at that point, you still need something like the BSD heuristics.

>AND carefully choosing the ISN).

If the collision happens for t < TIME-WAIT_state, it doesn't matter 
to choose the "older" port, becuase the connection will fail, anyway.

>I think if we can find a solution that attempts to be both mostly 
>full-proof in finding unique ports, and
>capable of reusing ports smartly when all are in use, this could 
>really be offered

Side question: Not that it is funny to talk about it, but: are there 
any IPRs on the stuff you describe in your draft?

Kind regards,

---

From: "Patrick Tate (ptate)" <[email-masked]>
Date: Thu, 10 Jul 2008 21:33:50 -0400

Fernando,

> Sent: Thursday, July 10, 2008 11:38 AM
> To: Patrick Tate (ptate); Anantha Ramaiah (ananth); [email-masked]
> Subject: RE: [Tsvwg] Effects of port randomization with TCP 
> TIME-WAIT state.
> 
> > > > > 3.3.4 from our draft you'd find the need for timers 
> or the like. 
> > > > > The problems you describe arise from the use of the 
> simple port 
> > > > > randomization algorithm.
> >
> >Just want to point out that even algorithms such as 3.3.3 
> and 3.3.4 can 
> >produce ports that will result in a TIME-WAIT collision if you have 
> >connections are long-lived.  Once the long-lived connections 
> start to 
> >go down and new ephemeral ports are created using one of these 
> >approaches, next_ephemeral could well be around the value used for 
> >these connections.
> 
> Yes, but this is also the case for the traditional BSD 
> algorithm, and that is the algorithm that has been in use in 
> most stacks for at least the last 20 years. At it has worked 
> fine even when using a very small ephemeral port range.
> 
> 
> 
> * Choosing port numbers incrementally is a cheap and 
> effective way to reduce the port number reuse frequency. This 
> has been what virtually all implementations have been doing 
> for the last 20 years or so.
> 
> * While it's great from the pov of interoperability, it's not 
> from the pov of security/obfuscation.
> 
> * No matter how you pick port numbers (whether with one of 
> the algorithms in our draft, or with one of the methods in 
> your draft), at the point in which your connection rate is 
> something like ephemeral_port_range/time_wait_state , you may 
> run into trouble. 
> * If you implement some mechanism that uses timers (as in any 
> of the methods you describe in your draft), you will still 
> face the problem I mentioned in the previous bullet when the 
> connection rate approaches 
> ephemeral_port_range/time_wait_state. Things *might* be 
> connections and the connection-rate is high enough that you 
> cycle through the whole port number space and have to reuse 
> some port numbers. However, the timers in your draft are 
> per-port (and not per-five-tuple), and thus that itself might 
> greatly increase the port reuse frequency in those scenarios 
> in which the connection requests are sent to more than one 
> peer (because it does not take advantage of the fact that 
> only the five-tuple, rather than the source port, has to be unique).

Good summary and points and I agree ...

> >this would be to find a way that is mostly full-proof for finding a 
> >port that should not result in the TIME-WAIT collision.
> 
> Clearly, collisions should be avoided. But no matter what you 
> do, if the connection-rate is high enough, collisions will 
> take place. And at that point, you still need something like 
> the BSD heuristics.
> 
> 
> 
> >to reuse from the set if you have run out of ports by carefully 
> >choosing the ISN (or perhaps finding the oldest of such ports as 
> >Anantha suggests AND carefully choosing the ISN).
> 
> If the collision happens for t < TIME-WAIT_state, it doesn't 
> matter to choose the "older" port, becuase the connection 
> will fail, anyway.
> 
> 
> 

No there are none.

-Patrick

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Jul 2008 14:12:36 -0700

Fernando Gont wrote:
...
> performs the active close), the situation is even better: you can reuse 
> the port number. Just be careful with the ISN you pick.

You can't reuse the port number just because you hold the TIME-WAIT. 
That state applies to connections initiated from either side of the 
connection.

Yes, you can pick a convenient ISN to avoid obvious overlap, but you're 
still risking accepting data from an old connection if the window wraps 
faster than the old data clears out.

> Now, even if you knew, once you have run out of port numbers, the most 
> sensible thing to do is to reuse the port numbers, carefully picking the 
> ISN for the new incarnation of the connection. (no matter whether the 
> socket in the TIME-WAIT state is local or remote).

When you run out of port numbers, you WAIT. That's what the WAIT in 
TIME-WAIT means.

Joe

---

From: Rick Jones <[email-masked]>
Date: Mon, 14 Jul 2008 15:10:12 -0700

> faster than the old data clears out.

Wasn't that "covered" by the assumptions on speed at which the window 
would wrap - which was one of the reasons we needed to add timestamps?

Anyhow, I thought that "picking a convenient ISN" was pretty much toast 
if one was actually doing "random" ISN - ie that the requirement to go 
to really random ISN's more or less meant that TIME_WAIT reuse was toast.

> When you run out of port numbers, you WAIT. That's what the WAIT in 

Well, when TIME_WAIT was defined did folks really think that systems 
would churn through port numbers as fast as they can/do now?  Unless 
they did I would have thought the WAIT in TIME_WAIT was all about 
waiting for the data to expire in the network and that running-out of 
port numbers wasn't really considered that seriously?

rick jones

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Jul 2008 15:23:45 -0700

...
> port numbers wasn't really considered that seriously?

When TIME-WAIT was defined, the Internet designers thought:

- 32 bits of address was enough
- 16 bits of source port was enough
- TCP option space was sufficient at 40 bytes
- TCP sequence numbers were sufficient at 32 bits
- TCP window sizes were sufficient at 16 bits
- IP IDs were sufficient at 16 bits

Plenty of things have changed since then. Some of the above have been 
patched (window scaling). Others have been ignored (IP ID, safely where 
fragmentation is prohibited, and unsafely otherwise).

Note that systems don't need to churn through port numbers as fast as 
they do right now; we don't need to associate TCP connections with 
transactions.

The WAIT was about how much you can trust TCP not to send the user data 
which is corrupted. If you use another protocol on top, e.g., SSL, you 
can reuse the WAITs as much as you want without impact. If not, however, 
you need to decide how long you expect things to float around in the 
net. In the beginnings of the Internet, things didn't float around all 
that long, but designers were more conservative (IMO).

Today, people are talking about resending the last packet a router sees 
to 'tickle' a connection into waking up when a train comes out of a 
tunnel - a tunnel which could be a few minutes long. IMO, the danger of 
relaxing these safety mechanisms increases as we forget/ignore what they 
were there to protect us from...

Joe

---

From: Fernando Gont <[email-masked]>
Date: Mon, 14 Jul 2008 19:29:10 -0300

At 06:12 p.m. 14/07/2008, you wrote:

>That state applies to connections initiated from either side of the connection.
>
>Yes, you can pick a convenient ISN to avoid obvious overlap, but 
>you're still risking accepting data from an old connection if the 
>window wraps faster than the old data clears out.

IIRC, this is a "may" in RFC 793, and has been deployed for at least 
10+ years in BSD-derived implementations.

That said, I don't think there's anything that prevents the window to 
wrap-around in less than MSL. COnsidering that timestamps are 
optional, you may already be hitting this issue.

>>picking the ISN for the new incarnation of the connection. (no 
>>matter whether the socket in the TIME-WAIT state is local or remote).
>

Most of the time, the collisions happen at the server. However, when 
they do happen, a new incarnation of the connection is created. That 
what all BSD-derived implementations have been doing for a long time, at least.

Kind regards,

---

From: Joe Touch <[email-masked]>
Date: Mon, 14 Jul 2008 15:46:25 -0700

Fernando Gont wrote:
> IIRC, this is a "may" in RFC 793, and has been deployed for at least 10+ 
> years in BSD-derived implementations.

The MAY is in 1122, not 793, and specifies that the ISN must be higher 
than the one in TIME-WAIT, not just random.

> wrap-around in less than MSL. COnsidering that timestamps are optional, 
> you may already be hitting this issue.

The TCP sequence number should never be able to wrap within 2MSL - IP ID 
uniqueness should have prevented that! And yes, we're already risking 
that in the way we already ignore IP ID uniqueness.

> what all BSD-derived implementations have been doing for a long time, at 

It all depends on what risks you want to take. One risk has to do with 
deciding what MSL is (how long do packets really recirculate?). Another 
has to do with sequence number wrap. They're all just risks.

I agree that we often take risks with deployed code. The question is 
whether we want to decide if that way of doing things is safe enough to 
recommend or not. Having used it for 10+ years doesn't make it safe - 
again, as we stretch how we use the Internet, we are more likley to step 
on the corner cases these safety mechanisms were intended to protect us 
from...

Joe

---

From: Fernando Gont <[email-masked]>
Date: Tue, 15 Jul 2008 03:09:57 -0300

At 07:46 p.m. 14/07/2008, Joe Touch wrote:

>>>You can't reuse the port number just because you hold the 
>>>TIME-WAIT. That state applies to connections initiated from either 
>>>side of the connection.
>>>
>>IIRC, this is a "may" in RFC 793, and has been deployed for at 
>>least 10+ years in BSD-derived implementations.
>
>The MAY is in 1122, not 793, and specifies that the ISN must be 
>higher than the one in TIME-WAIT, not just random.

I never said it should be random. For instance, that's the whole 
point of using RFC1948, rather than a simple randomization scheme.

>>That said, I don't think there's anything that prevents the window 
>>to wrap-around in less than MSL. COnsidering that timestamps are 
>The TCP sequence number should never be able to wrap within 2MSL - 
>IP ID uniqueness should have prevented that! And yes, we're already 
>risking that in the way we already ignore IP ID uniqueness.

Agreed. However, there has never been any implementation which 
actually throttles the transfer rate to satisfy those assumptions.

>>Most of the time, the collisions happen at the server. However, 
>>when they do happen, a new incarnation of the connection is 
>>created. That what all BSD-derived implementations have been doing 
>>for a long time, at least.
>
>It all depends on what risks you want to take. One risk has to do 
>with deciding what MSL is (how long do packets really recirculate?). 
>Another has to do with sequence number wrap. They're all just risks.

Agreed. FWIW, I'm not sure if it's possible/easy to arrive to some 
MSL value. We should be able to bind not only the maximum amount of 
time that packets can spend in routers, but also the maximum amount 
of time they can spend on the wire.

>whether we want to decide if that way of doing things is safe enough 
>to recommend or not. Having used it for 10+ years doesn't make it 
>safe - again, as we stretch how we use the Internet, we are more 
>likley to step on the corner cases these safety mechanisms were 
>intended to protect us from...

Agreed. Let's just say this BSD heuristic has been useful to kick the 
problem forward for sometime, adn has allowed (yes, with risks) a 
high connection-request rate that other wise might have not worked. 
However, as mentioned in my other e-mail, regardless of this 
BSD-hack, the sequence number might simply wrap within 2MSL as a 
result of the data transfer rate. This scenario will probably become 
more and more common than that of the BSD-hack killing the TIME-WAIT 
state of the prvious incarnation of the connection.

Kind regards,