Subject: Re: [tcpm] TCP zero window timeout?

From: Mahesh Jethanandani <[email-masked]>
Date: Fri, 25 Aug 2006 17:00:13 -0700

Jamshid,

Looking at draft-ietf-tcpm-tcp-uto it appears that the draft is 
specifically looking at the question of disconnection in the network. It 
also applies to retransmission timer.

The situation I was referring to is a little different and applies to 
persist timer. In our situation the client stops reading data. These 
clients are machines out in the Internet and as such the server has no 
control over their behavior. So while there is unacknowledged data, it 
is not that the client is not acking any data. It is responding to the 
probe but that it continuously advertises a window of zero.  There is 
currently to my knowledge no timeout for this state for the server. This 
can manifest itself as a DOS situation if there are several such 
connections where the server is forced to hold data.

We are suggesting a solution that allows the server to get out of this 
situation by applying a upper bound on the duration of the persist 
state. Note, it is not the default behavior for TCP. The default 
behavior is still the same. The user/administrator has to explicitly 
turn it on for the server to close the connection and free the resources 
in case it is believed that it is under attack.

Fernando Gont wrote:

> At 13:24 21/07/2006, Mahdavi, Jamshid wrote:
>
>> What is the status of draft-eggert-tcpm-tcp-abort-timeout-option-01?  It
>> may be of some use in situations like this.  I've recently seen another
>> scenario where this would be useful, so I'd be interested in seeing that
>> draft reposted...
>
>
> It was merged with draft-gont-tcpm-tcp-auto-option into 
> draft-ietf-tcpm-tcp-uto.
>
> The latest revision is draft-ietf-tcpm-tcp-uto-03.txt, available at 
> the usual places (e.g., 
> http://www.ietf.org/internet-drafts/draft-ietf-tcpm-tcp-uto-03.txt).
>
> Feedback is more than welcome. ;-)
>
> Kindest regards,
>
> -- 
> Fernando Gont
> e-mail: [email-masked] || [email-masked]
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
>
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Fri, 25 Aug 2006 19:48:08 -0700

Wouldn't this just result in the DOS attacker ACKing one byte at a time
to prolong the connection needlessly?

I.e., what's the point of putting this behavior inside TCP, vs. having
the server give up on a connection after what it considers a reasonable
time?

Joe

---

From: Fernando Gont <[email-masked]>
Date: Sat, 26 Aug 2006 07:13:56 -0300

At 21:00 25/08/2006, Mahesh Jethanandani wrote:

>The situation I was referring to is a little different and applies 
>to persist timer. In our situation the client stops reading data. 
>These clients are machines out in the Internet and as such the 
>server has no control over their behavior. So while there is 
>unacknowledged data, it is not that the client is not acking any 
>data. It is responding to the probe but that it continuously 
>advertises a window of zero.  There is currently to my knowledge no 
>timeout for this state for the server. This can manifest itself as a 
>DOS situation if there are several such connections where the server 
>is forced to hold data.

I'd argue that this should be handled by an application-level timer.

The problem with the scenario you point out is that there will always 
be one more way to do the same thing.

If we're talking about connections wasting resources, there are many 
protocols (POP3, SMTP) in which after the initial greeting by the 
server, the client is supposed to go ahead. In all those cases, the 
client could just sit there. And only an application-layer timeout 
would help you.

I'm not sure how many servers implement this type of 
application-layer timer. But there are some (Apache) that I have 
checked, and do.

Nevertheless, I'm interested in the behaviour you described. Is it 
supposed to be malicious activity?

Kindest regards,

---

From: MURALI BASHYAM <[email-masked]>
Date: Sat, 26 Aug 2006 16:05:45 -0700 (PDT)

Fernando

I collaborated with mahesh on this, so let me try
making the case for it a little better.

The problem was found on a TCP proxy, which does not
have any applicaton awareness. In fact application
awareness is not a goal in that environment. Hence the
TCP level solution.

Having said that, i'd say this timeout is in the same
spirit as the upper bound on the retransmit mechanism
of TCP. TCP could indefinitely retransmit and have the
application timeout too, correct? The problem is that
TCP has a persist state which can potentially exist
infinitely long  and which lends itself to abuse by a
malicious peer. Just based purely on this
consideration, don't you think TCP should have a
mechanism that reduces the impact of this problem by
either limiting the number of probes (a la retransmit
timeout) or the duration. And the application can turn
it on per connection via a socket option and set a
value which makes sense for that application, so there
is per application per connection control anyway.

Murali

> https://www1.ietf.org/mailman/listinfo/tcpm

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

---

From: Joe Touch <[email-masked]>
Date: Sat, 26 Aug 2006 19:58:43 -0700

> awareness is not a goal in that environment. Hence the

Application-aware proxies are one thing (and they would be able to
adjust fine here). TCP proxies for splice-like, transport-only gateways
are already known to break so many things it's not clear that
considering that as a special case is a good argument for modifying TCP.
It's a better argument not to use TCP that way.

> malicious peer. 

TCP is not intended to be robust to security attacks. A peer that can
establish a TCP connection is already presumed to be a non-attacker at
the TCP level, IMO.

JOe

---

From: "Kuthonuzo Luruo (STSD)" <[email-masked]>
Date: Mon, 28 Aug 2006 18:02:15 +0530

> is per application per connection control anyway.

Several OSes do include a socket option SO_SNDTIMEO, that can be set by
an application to specify the amount of time a transmit function blocks
when flow control prevents the transmission of data. 

-thonuzo

---

From: Ted Faber <[email-masked]>
Date: Mon, 28 Aug 2006 10:36:59 -0700

> awareness is not a goal in that environment. Hence the

(As just a participant)

Why get TCP involved here?

If a proxy is running out of memory and it has connections that are
largely unused, it should close them.  The proxy is at the exact right
spot to keep track of the connections and their usage.  It can tell
which connections are wasting its resources and it knows what it
considers bad behavior of a connection.  When proxy resources get
scarce, close the connections that are behaving the worst - have been
holding the same proxy resources a long time, for example.

None of that requires application-awareness or TCP's complicity.

And even if it did, TCP enables many applications.  To a first
approximation, increasing TCP's complexity increases its fragility.
Changes to TCP should benefit more than one application before they're
standardized, IMHO.

-- 
Ted Faber
http://www.isi.edu/~faber           PGP: http://www.isi.edu/~faber/pubkeys.asc
Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Fernando Gont <[email-masked]>
Date: Mon, 28 Aug 2006 17:50:48 -0300

At 20:05 26/08/2006, MURALI BASHYAM wrote:

>application timeout too, correct?

I disagree. In the case of the RTO, you don't have any hints of the 
TCP segments getting to the remote TCP endpoint. Hence, after some 
number of retransmission, you don;t have much else to do than to 
abort the connection.

In the case of zero window, as far as TCP is concerned, there's 
nothing wrong with an endpoint advertising a zero window. And you do 
know that packets are getting to the remote TCP endpoint (that's why 
you are getting the advertisements, after all).

>The problem is that

This is the same thing as saying that there should be an "idle 
connection timeout", because a connection can be opened, and no data 
needs to be transmitted on the connection for the connection to be kept open.

That said, some upper limit on the number of window-probes might not 
hurt, but then you get into which value to use.
If too small, you may kill legitimate users. If too long, it may be useless.

And I doubt there really is something in between.

Kindest regards,

---

From: MURALI BASHYAM <[email-masked]>
Date: Mon, 28 Aug 2006 15:59:43 -0700 (PDT)

> Why get TCP involved here?

Because TCP's requirement to persist indefinitely
creates the problem and only TCP is aware of how long
the peer has been doing zero window offering.

> holding the same proxy resources a long time, for

This does not take into account the fact that
connections are persisting and probing for peer's zero
window which is fundamental to the discussion.

> application before they're

A TCP sender which is simply trying to probe the peer
for an indefinitely long amount of time certainly
stops helping the application or the system once
beyond some amount of time threshold. The threshold
may be different for different  applications but for a
given application and a system the threshold surely
exists.  

Murali

---

From: MURALI BASHYAM <[email-masked]>
Date: Mon, 28 Aug 2006 16:15:07 -0700 (PDT)

Fernando

Today there is nothing that prevents a client side
application from simply stopping to read the TCP
receive socket buffer and causing the offered window
to go down to 0, and thus causing the sender to hold a
large send queue worth of data and probe forever. If
this is done by a large number of clients against the
same server, u have a distributed DOS attack on that
server. We have seen this in practice. 

To answer an earlier question u had raised, the
application cannot timeout on this connection because
it does not know when the connection enters and leaves
the persist state, only TCP knows that. The
application can definitely decide the timeout value,
but TCP needs to implement the timer because only it
is aware of the state of the peer. 

More comments inline.

> number of retransmission, you don;t have much else

I agree with you on that, but what i want to borrow
from that timer is the aspect of limiting the amount
of time trying to retransmit, which is what i meant.

> you are getting the advertisements, after all).

And i  am not disagreeing with you on that, let's
differentiate the legitimate ones from the bad ones or
minimize the impact of the bad ones.

> connection to be kept open.

An idle connection does not consume any buffer
resources, whereeas here the connection is holding up
precious buffer resources.

> If too small, you may kill legitimate users. If too
 
I agree, but in the event of a DOS like scenario, this
provides the administrator with a tool to control the
impact of these bad clients on the system, and he can
come up with a value which certainly does not impact
legitimate probes. In these scenarios, any limit is
better than infinite :-).

Murali

---

From: Mahesh Jethanandani <[email-masked]>
Date: Mon, 28 Aug 2006 16:49:48 -0700

> you are getting the advertisements, after all).

The problem with the client advertising a zero window is that it lends 
itself to a DOS attack. The problem is particularly bad if you are 
dealing with thousands of connections each of them chewing up connection 
blocks and TCP data buffers. That is what happened in the particular 
customer scenario.

> needs to be transmitted on the connection for the connection to be 

But you would agree that a large number of idle connections without a 
idle timeout is a problem for any TCP implementation.

> If too small, you may kill legitimate users. If too long, it may be 

If we agree that a limit is required on the number of probes then we can 
move to discussing how to implement that, what its default value should 
be and whether that value can be changed by the user etc.

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 28 Aug 2006 16:50:49 -0700

MURALI BASHYAM wrote:
> application from simply stopping to read the TCP receive
> socket buffer and causing the offered window to go down to 0,
> and thus causing the sender to hold a large send queue worth
> of data and probe forever. If this is done by a large number
> of clients against the same server, u have a distributed DOS
> attack on that server. We have seen this in practice.
> 
> To answer an earlier question u had raised, the application
> cannot timeout on this connection because it does not know
> when the connection enters and leaves the persist state, only
> TCP knows that. The application can definitely decide the
> timeout value, but TCP needs to implement the timer because
> only it is aware of the state of the peer.
> 

True, but why does TCP need any wire protocol modifications to
implement such a timeout locally?

---

From: MURALI BASHYAM <[email-masked]>
Date: Mon, 28 Aug 2006 17:04:02 -0700 (PDT)

--- Caitlin Bestler <[email-masked]> wrote:

> receive
> application

It does not need any wire protocol modifications, but
the intent of this discussion is whether it is
worthwhile to document this at all and if so as a good
practice or  as an informational RFC so that
implementers are aware of the issue and potential
solutions and so on.

Murali

---

From: Fernando Gont <[email-masked]>
Date: Mon, 28 Aug 2006 17:43:29 -0300

>when flow control prevents the transmission of data.

This is not exactly true.

IIRC, SND_SNDTIMEO is a timeout on the relevant socket call (send(), 
write(), etc.), and not on actual data delivery to the remote TCP endpoint.

As long as there's is enough free space in the socket send buffer, 
write()/send() won't block, and therefore the SO_SNDTIMEO timeout won;t apply.

(Yes, if the TCP sender is sending bulk data, at some point the send 
buffer will most lilely fill up, and this timeout will kick in....)

Kindest regards,

---

From: Ted Faber <[email-masked]>
Date: Tue, 29 Aug 2006 09:57:56 -0700

> the peer has been doing zero window offering.

TCP is completely unaware of how long a client has been offering a 0
receive window.  You'd have to change it to keep count of that.  That's
what you're proposing, correct?  

As an aside, RFC1122 says pretty unambiguously that both holding a zero
window and probing it regularly are intentionally supported.

> window which is fundamental to the discussion.

The resource your proxy is running out of is CPU to do window probes???
If so, that's very surprising.  If not, can you tell me what your
concern is?

Looking at RFC1122 Section 4.2.2.17, it's hard to imagine a less CPU-
intensive way to deal with window probing than the exponential backoff
algorithm suggested there.

> given application and a system the threshold surely

Without getting too philosophical, the designers of TCP did explicitly
support connections that lock out transmission via a zero window size.

---

From: MURALI BASHYAM <[email-masked]>
Date: Tue, 29 Aug 2006 11:13:14 -0700 (PDT)

> what you're proposing, correct?  

Correct.

> window and probing it regularly are intentionally

It does, and i dont understand the rationale of
probing *infinitely* as long as the client is
acknowledging the probe with a zero window. It's
possible that the authors did not have the potential
of a DOS attack when they created this mechanism. In
the scenario that i am talking abt these probes were
sometimes running for hours together and such
connections were ultimately starving out other
legitimate users traffic.

> If so, that's very surprising.  If not, can you tell

The resources in question are connections and buffers.
Here we are talking potentially huge numbers of them
(100000 connections and even if each connection holds
1 buffer, that's a lot of buffer memory). A mechanism
to reclaim these resources  would have to take into
account the duration of the persist state of the
connections, it can't be done blindly.

> support connections that lock out transmission via a

I think the crux of the issue here is that this is
left to the client side application control which
means open game for hackers.

Murali

---

From: "Caitlin Bestler" <[email-masked]>
Date: Tue, 29 Aug 2006 11:33:56 -0700

MURALI BASHYAM wrote:
 
>> 
> 
> I think the crux of the issue here is that this is left to
> the client side application control which means open game for hackers.
> 
> Murali

I think it is a given that a server application can terminate
the connection to any client if the behaviour of that client
means that the server no longer wishes to maintain the connection.

I am not aware of any RFC that would impair the ability of
the server application to interact with the local TCP stack
to achieve these results.

Is there a specific RFC requirement that you believe impairs
creation of local interfaces to achieve the results you desire?
In general, if the application layer has the right to do something
it has the right to delegate that authority to a lower layer on
the local stack and doing so will contradict the wire protocol.

---

From: John Heffner <[email-masked]>
Date: Tue, 29 Aug 2006 14:41:25 -0400

I was looking at a similar problem a while ago as a result of a 
conversation with Stanislav Shalunov about his netkill 
(http://www.internet2.edu/~shalunov/netkill/) attack tool.  The netkill 
problem is different than the zero window problem (and simpler for the 
attacker, requiring less state), but the end result is essentially the 
same: the sender is forced to maintain data in the send buffer for a 
large number of connections, running the server out of memory.

Thinking about possible solutions, I didn't like the strictly TCP or 
strictly application solutions, for reasons already described on this 
list.  My favored approach was to write a daemon that accesses TCP state 
through instrumentation, applies a policy if memory is running low, and 
kills connections as appropriate by using the deleteTCB(12) state 
defined in the TCP MIB.  This does not require any additional protocol 
standardization.

I wrote a simple proof-of-concept python script (below) that utilizes 
the Web100 implementation of the TCP ESTATS MIB.  What it does 
essentially is if the system TCP send buffer memory is over some 
threshold value (100 MB), then it kills all connections that are over 
their fair share, and haven't made any forward progress in the last 
second.  There's clearly room for improvement, like more complicated 
policy (protect or favor certain applications, etc.), but it does defend 
against netkill, and should work for the zero-window attack as well.

   -John

import Web100
import time

agent = Web100.Web100Agent()

THRESH = 100 * 1024 *1024

omem = 0
conns = {}
while True:
     cmem = 0

     cl = agent.all_connections()
     for c in cl:
         state = c.read('State')
         try:
             old = conns[c.cid]
             if state == 1:
                 del(conns[c.cid])
             else:
                 cur = c.readall()

                 cmem = cmem + cur['CurAppWQueue']

                 if omem > THRESH and \
                    cur['SndUna'] == old['SndUna'] and \
                    cur['CurAppWQueue'] > THRESH / len(conns):
                     print("Deleting connection %d."%c.cid)
                     c.write('State', 12)

                 conns[c.cid] = cur

         except:
             # New connection.  Ignore if it's already closed.
             if state != 1:
                 conns[c.cid] = c.readall()
                 cmem = cmem + c.read('CurAppWQueue')

     omem = cmem
     time.sleep(1)

---

From: MURALI BASHYAM <[email-masked]>
Date: Tue, 29 Aug 2006 11:54:44 -0700 (PDT)

--- Caitlin Bestler <[email-masked]> wrote:

> left to

The application is not aware of the client's behaviour
here and as far as he is concerned the data is out of
his control. 

> to achieve these results.

Are u suggesting that the application query TCP to
determine that the connection is stuck in persist
state and how much of the data has been sent or not
and so on? That sounds a little roundabout and
complicated...
> the local stack and doing so will contradict the

The application layer does not have enough information
to take the corrective actions here.

Murali

---

From: "Caitlin Bestler" <[email-masked]>
Date: Tue, 29 Aug 2006 12:34:15 -0700

MURALI BASHYAM wrote:
>> I think it is a given that a server application can terminate the
>> connection to any client if the behaviour of that client means that
>> the server no longer wishes to maintain the connection.
>> 
> 
> The application is not aware of the client's behaviour here
> and as far as he is concerned the data is out of his control.
> 
>> I am not aware of any RFC that would impair the ability of the server
>> application to interact with the local TCP stack to achieve these
>> results.
> 
> Are u suggesting that the application query TCP to determine
> that the connection is stuck in persist state and how much of
> the data has been sent or not and so on? That sounds a little
> roundabout and complicated...

And/or the application convey an application specific tolerance
for such a state through a local interface.

If the behavior can be corrected through a local interface,
and there is no IETF language barring or discouraging such
an interface, then why is this something that TCPM should
consider?

At first glance, John Hefner's post would seem to show that
the required information is already available.

The nature of the Denial of Service attack presented does
not seem to require real-time defenses and definitely does
not present a strong case for any network element modifying
their "fast path" handling of packets.

---

From: Mahesh Jethanandani <[email-masked]>
Date: Tue, 29 Aug 2006 14:29:00 -0700

>for such a state through a local interface.
We agree that application can help TCP by suggesting how tolerant  it 
wants to be.

>an interface, then why is this something that TCPM should
Information of how long the connection has been stuck in persist state, 
how many retry probes have been made and whether each of them has 
resulted in a zero window is all in TCP. So TCP(m) has to be involved. 
Who actually issues the reset on the connection is a implementation 
level detail which we can get into once we agree that we have a DOS 
situation.

>their "fast path" handling of packets.
If we accept that it is a DOS attack, then the problem needs to be 
addressed. Our suggestion only says that today's behavior of infinite 
TCP probes in persist state lends itself open to a DOS and so the probes 
should have the option to be capped with a timeout.
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Tue, 29 Aug 2006 16:49:05 -0700

MURALI BASHYAM wrote:
> It's
> of a DOS attack when they created this mechanism.

TCP does not itself defend from DOS attacks. If you want to prevent
that, run connections over IPsec.

...
> connections, it can't be done blindly.

TCP isn't there to clean up state. If old connection state is
interfering with new connections, and the connection isn't making
progress, the application layer (the layer that runs the buffers) can
tell. That's an application-layer timeout.

Joe

---

From: Ted Faber <[email-masked]>
Date: Wed, 30 Aug 2006 08:22:11 -0700

> legitimate users traffic.

The probes come a long time apart after a little while - exponential
backoff.

If you're running out of buffer space, abort the connections.

> connections, it can't be done blindly.

But that state is easily kept track of by the application.  Close the
ones that are using the most space and have made the least progress.
Even if your implementation doesnt give you access to the full state a
conformant TCP stack does (RFC 793 p. 49) gathering that information is
pretty easy.  I don't know what your environment is, but most have some
way to get this or infer it.

Once the app is short of space and knows which connections it likes
least, it can abort them and reclaim all the resources.

> means open game for hackers.

I agree with your premise, but not your conclusion.

I don't understand why you're against doing this in the app, but I don't
think I understand your whole problem either.  I suggest writing a draft
and giving us the full picture.

---

From: Joe Touch <[email-masked]>
Date: Wed, 30 Aug 2006 09:19:19 -0700

Ted Faber wrote:
...
> and giving us the full picture.

I'd rather see succinct answers to the alternatives suggested here that
motivate a TCP-level solution first.

IDs can easily themselves be a DOS attack on IETF resources.

Joe

---

From: Ted Faber <[email-masked]>
Date: Wed, 30 Aug 2006 09:23:46 -0700

> motivate a TCP-level solution first.

There are six people exchanging mail about a problem and a solution that
don't seem well defined to me.  IMHO agreeing on what we're talking
about will simplify that discussion.
> IDs can easily themselves be a DOS attack on IETF resources.

Or a necessary step forward.

---

From: Joe Touch <[email-masked]>
Date: Wed, 30 Aug 2006 09:29:00 -0700

Ted Faber wrote:
> don't seem well defined to me.

So far here's what I see:

	the proposers consider "application" to be the end-to-end
	user program, which may have a data format unavailable to
	the proxy

	TCP considers "application" to be the controlling program
	that manages the TCP connection, e.g., the user program
	at the endpoints, or the system inside the proxy

The system inside the proxy may not know what the user is doing at the
endpoints, but it DOES know whether its buffers are empty or full, and
whether the connection is making progress.

That system can easily decide when to shut connections down if progress
isn't being made.

---

If an ID described the details of the proxy system such that the above
were NOT possible, then that's fine.

If the ID makes assertions about "proxies out there" and how they
"might" work, then an ID is not productive.

> IMHO agreeing on what we're talking

Yes, but an ID isn't necessarily a step in that direction. A one-page
description of the problem is more productive and less work for
everyone. If one page is insufficient, a one page explanation of WHY one
page is insufficient seems a more useful step forward.

Joe