Subject: [tcpm] Is this a problem?

From: Mahesh Jethanandani <[email-masked]>
Date: Mon, 29 Oct 2007 14:47:37 -0700

Folks,

We have documented a case of HTTP servers that are prone to resource starvation with the use of a small user level program. The program does not require any special privileges or changes in the kernel. The user level program on the client opens a connection to a HTTP server, sends a GET request for a large file (larger than the advertised window of the client) but never reads the response.

Three well-known, public sites were tested for this vulnerability. The two most common HTTP servers, Apache and IIS were the target. While one site had put mitigation technique in place, the others had none. With the latter two we were able to hold connections in ESTABLISHED state for days. The former site had a mitigation in place with a fixed timeout of 11 min., which was easy to guess and work around.

We (the authors) believe that this is a huge problem. What do you folks feel?

Previous responses to this documentation has been that it is a application problem. It is clear from our experimentation that most HTTP servers (and FTP too) have not implemented any mitigation techniques. We believe that this problem exists across the whole range of TCP based applications prevalent on the internet, although our experiments were limited to the web application. Where applications have tried to put mitigation techniques in place, workaround has been easy. This is mainly because applications do not have the same amount of visibility as TCP does on the state of the connection.

If you are interested in reading the draft, you will find it here <http://www.ietf.org/internet-drafts/draft-mahesh-persist-timeout-02.txt>.

/mahesh

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: "Caitlin Bestler" <[email-masked]>
Date: Mon, 29 Oct 2007 18:04:00 -0400

This is indeed a problem. I'm just not convinced it's a transport
problem.

Only the application layer has enough information to know when a
client's

behavior is abusive and/or suspicious. The transport layer should remain

militantly ignorant of these matters.

---

From: Joe Touch <[email-masked]>
Date: Mon, 29 Oct 2007 15:42:41 -0700

Mahesh Jethanandani wrote:
> Folks,
> 
> We have documented a case of HTTP servers that are prone to resource
> starvation with the use of a small user level program. The program does
> not require any special privileges or changes in the kernel. The user
> level program on the client opens a connection to a HTTP server, sends a
> GET request for a large file (larger than the advertised window of the
> client) but never reads the response.
> 
> Three well-known, public sites were tested for this vulnerability. 
> The two most common HTTP servers, Apache and IIS were the target.
> While one site had put mitigation technique in place, the others had
> none. With the latter two we were able to hold connections in
> ESTABLISHED state for days. The former site had a mitigation in place
> with a fixed timeout of 11 min., which was easy to guess and work
> around.
> 

I agree with Caitlin and your previous responses; this is an application
problem.

> Previous responses to this documentation has been that it is a
> application problem. It is clear from our experimentation that most HTTP
> servers (and FTP too) have not implemented any mitigation techniques. We
> believe that this problem exists across the whole range of TCP based
> applications prevalent on the internet, although our experiments were
> limited to the web application. Where applications have tried to put
> mitigation techniques in place, workaround has been easy. This is mainly
> because applications do not have the same amount of visibility as TCP
> does on the state of the connection.

Applications know when a connection hasn't closed. They also know when a
write would block due to insufficient socket resources. Either or both
of these provide sufficient visibility to avoid the problem entirely.

As you note, applications can solve this easily. Since it's their
obligation to do so, and it's easy for them to do so, there's no reason
to complicate the transport layer with this responsibility.

Furthermore, does the propose solution solve the DOS problem? Aren't
there other ways to keep a source stalled? (i.e., what about continued
SACKs? or repeated ACKs indicating a lost segment? at the very least, we
could ACK a byte at a time, sending 3-4 duplicate ACKs for each byte,
which would keep the sender from opening its window and stall things a
lot...).

Joe

---

From: Mahesh Jethanandani <[email-masked]>
Date: Mon, 29 Oct 2007 17:19:43 -0700

Caitlin,

Caitlin Bestler wrote:
>
> This is indeed a problem. I'm just not convinced it's a transport problem.
>
>  
>
> Only the application layer has enough information to know when a client's
>
> behavior is abusive and/or suspicious.
>
Applications if anything have less knowledge than TCP on why a 
connection is not making progress. In this particular case, HTTP could 
not make a distinction between a connection not making progress because 
of congestion in the network or because the client is advertising zero 
window. And yet all Apache can do is to reset the connection after a 
fixed amount of time. Again, HTTP may know how many of its connections 
are in the stuck state, it will not know how many FTP connections are 
seeing the same issue or how seriously constrained the system is.

We have found that a fixed timeout also does not work. It is easy to 
guess and work around as we did with our client program.

The solution has to be adaptive to the condition which in this case 
involves TCP and applications have little knowledge of the state of all 
of TCP connections.
>
> The transport layer should remain
>
When it impacts TCP's ability to function, I can hardly see that being 
ignorant will help.
_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Mon, 29 Oct 2007 22:15:55 -0700

Mahesh Jethanandani wrote:
> Caitlin,
> 
> Caitlin Bestler wrote:
>>
>> This is indeed a problem. I’m just not convinced it’s a transport problem.
>> Only the application layer has enough information to know when a client’s
> window. 

If an application - or host - has limited resources, it doesn't matter
why a connection isn't making progress.

> And yet all Apache can do is to reset the connection after a

There is no rule that TCP connections "see" each other per se; from
within a single connection, an implementation could be able to access
only a single connection's state.

And, ultimately, a system is constrained when it runs out of processor
or memory - which have nothing to do necessarily with the connections in
progress. I.e., an application can ask the OS - or the OS can tell the
app - when such limitations come into play anyway.

> When it impacts TCP's ability to function, I can hardly see that being

You're declaring that someone who sends a zero receive window longer
than you expect to be noncompliant. There's nothing in the spec that
says that a TCP connection has to make forward progress in any fixed
timescale - even based on RTTs. This has allowed TCP to proceed over
shared links where less than one segment succeeds per RTT - e.g.,
congested transatlantic links. My concern is that this modification will
disable correct behavior in the corner cases.

Not all TCP connections must - or should - work only for the common cases.

Joe

---

From: Mahesh Jethanandani <[email-masked]>
Date: Mon, 29 Oct 2007 23:46:00 -0700

> of these provide sufficient visibility to avoid the problem entirely.
Knowing those two facts only gives applications some of the information. 
It still would not know why the write is blocked? You agree that 
penalizing legitimate connections is a broken implementation. How does 
application then differentiate between a legitimate connection that is 
slow and one that is malicious (by advertising a zero window for an 
extended period of time)?

It is important to understand that a slow legitimate connection is a 
behavior that is almost entirely contained within TCP but the client 
sending zero window can be a case of a malicious user level program 
(read externally triggered attack) not reading data. Looking into the 
TCP connection state allows us to differentiate between the two 
different conditions and treat the connections fairly.
> to complicate the transport layer with this responsibility.
The reality is that few applications if any implement any mechanism to 
detect a connection that is stalled in such a manner. Our experiment 
shows that  the most commonly used application (HTTP) used on some of 
the biggest web sites are vulnerable today holding on to connections in 
EST state for days. Yes, they should implement a mechanism, but can TCP 
count on it. What is it supposed to do if applications do not clear 
connections in a timely manner.

> which would keep the sender from opening its window and stall things a
You are talking about a malicious TCP implementation that would require 
special permissions in a DDoS scenario. What we used was a simple user 
level program that required no special privileges to run.

---

From: Joe Touch <[email-masked]>
Date: Tue, 30 Oct 2007 06:32:38 -0700

Mahesh Jethanandani wrote:
> penalizing legitimate connections is a broken implementation. 

My point is that your solution does this.

> How does

Since it never can, it shouldn't try. It depends on the application.
That's why it's the application's decision as to when to give up on a
client, not TCP's.

> (read externally triggered attack) not reading data. 

I agree that sending a zero window CAN be an attack. Unfortunately, it
CAN be legitimate behavior.

Since you agree we can't decide between the two at the TCP layer, we
should not fix this at that layer either.

> Looking into the

You are asserting that "all zero-window receivers are attackers". This
is incorrect, and leading you to the conclusion that TCP should fix this.

A legitimate receiver could have a backed-up application - i.e., a slow
processor, more pressing other applications, etc. - which could cause it
to advertise a zero window for an arbitrary time.

There is no requirement of timeliness in TCP data processing. When data
arrives at the receiver, if it backs up, the receiver could send a zero
window. That condition could legitimately persist a long time - I gave a
real example before (a backup device waiting for a human to insert a new
DVD).

> detect a connection that is stalled in such a manner.

I agree. That doesn't make it TCP's responsibility.

> Our experiment

TCP is the servant of the application, not its master. If the
application doesn't clear connections TCP should - and must - sit and wait.

> level program that required no special privileges to run.

Yes, but as I've shown, your case is indistinguishable from legitimate
behavior. Let's not 'fix' that.

Joe

---

From: Florian Weimer <[email-masked]>
Date: Tue, 30 Oct 2007 15:14:19 +0100

* Mahesh Jethanandani:

> Three well-known, public sites were tested for this vulnerability. The
> two most common HTTP servers, Apache and IIS were the target. While
> one site had put mitigation technique in place, the others had

It's a problem, but not at the TCP layer.

SSH connections, for instance, can legitimately last for days in
ESTABLISHED state without any traffic being exchanged.

For HTTP, it may be helpful to have some kind of information regarding
the TCP state to mitigate the DoS potential, but I fear that it's too
easily manipulated by the attacker to be useful.  Even if it's not, it's
really a sockets API issue.

---

From: John Heffner <[email-masked]>
Date: Tue, 30 Oct 2007 14:58:58 -0400

> We have documented a case of HTTP servers that are prone to resource starvation with the use of a small user level program. The program does not require any special privileges or changes in the ker…

This problem has been well documented for at least seven years.
http://shlang.com/netkill/
http://shlang.com/netkill/20000421-netkill-bugtraq-announcement.txt

> We (the authors) believe that this is a huge problem. What do you folks feel?

I personally believe it is a problem, but the absence of wide-spread 
attacks using this technique leads me to believe that other attacks are 
still more effective.  All the same, I would like to see more widespread 
defense against it.

> Previous responses to this documentation has been that it is a application problem. It is clear from our experimentation that most HTTP servers (and FTP too) have not implemented any mitigation tec…

I think nearly all responses (including mine) have been of the opinion 
that this is not a transport problem.  However, I think this is not 
necessarily an application problem, either.  Since the contended 
resource is system memory, the best place to implement a fairness policy 
is in the operating system -- possibly but not necessarily in the kernel 
(for OS's where such a distinction applies).

   -John