Subject: Re: [tcpm] Is this a problem?

From: Joe Touch <[email-masked]>
Date: Mon, 29 Oct 2007 22:59:43 -0700

MURALI BASHYAM wrote:
> I agree with Caitlin and your previous responses; this is an
>  application
> problem.
> 
> So we are placing the responsibility for protecting system-wide TCP resources such as buffers
> and connections on all the TCP based applications out there? 

Yes - applications can always start extra TCP connections, and can
override default socket sizes, so those resources are already under
application control. They are managed by the OS as a whole, i.e., when
the OS runs out, the application gets that feedback, and reacts accordingly.

> Which "application"
> are you referring to here? BGP? NFS ? CIFS? HTTP? SMTP? FTP? The entire application
> community? Potentially several or all of these apps are vulnerable. How do we bring all these 
> protocols into a single agreeable framework and  accomplish the right thing? 

Any application can misbehave. Any application can hold off receiving
data it has asked for. If the sender side application cares, it cuts off
the connection. If it doesn't, it leaves the connection in place.

TCP doesn't read minds.

...
> Applications cannot discriminate between a connection that's stalled due to network
> congestion and a connection that's being stalled by a malicious receiver who has closed
> his receive window forever. And for how long the connection has been in in this stalled state etc.

Applications shouldn't discriminate between these. First, there's no
difference between a connection that's being stalled maliciously and one
that's being stalled legitimately. There's no requirement that TCP not
be stalled for a particular interval. You'd be disabling TCP's ability
(feature) to work over such environments arbitrarily if you change this.

> And what about multi-application servers? Surely each application here is running as an
> independent process and indvidually they don't have the system wide awareness that TCP has...

Nor should it. First, as I noted in other mail, TCP doesn't necessarily
have system-wide information - that's an *assumption*. You're talking
about the kernel making decisions, not TCP. TCP could be implemented in
user space, or in the kernel in a way that it is isolated from
information about other connections. That's another reason this isn't
TCP's job.

> As you note, applications can solve this easily. Since it's their
> obligation to do so, and it's easy for them to do so, there's no reason
> to complicate the transport layer with this responsibility.
> 
> Applications can implement a fixed or even an adaptive timeout, but unless they can make
> the above distinction i am pointing out, they will be knocking out legitimate (albeit slow) 
> connections, i.e there will be a fairness issue. 

And they'll still be knocking out legitimate (albeit slow) receivers
that have a good reason to send a zero window for some period of time -
like while you insert a new DVD to backup to.

> Applications do not have visibility into TCP connection
> sender's state to solve the fairness issue. 

Visibility into the state of a single connection does not solve this
issue. You simply know that the receiver is not accepting data at this
time - this is a feature of TCP. It's not a bug.

> A solution which penalizes legitimate connections is clearly a broken one.

Agreed!

> Furthermore, does the propose solution solve the DOS problem? Aren't
> there other ways to keep a source stalled? (i.e., what about continued
> SACKs? or repeated ACKs indicating a lost segment? at the very least,
>  we
> could ACK a byte at a time, sending 3-4 duplicate ACKs for each byte,
> which would keep the sender from opening its window and stall things a
> lot...).

> How do you propose to carry out these attacks without changing the client side TCP stack in the 
> windows or linux kernel? The vulnerability which we have demonstrated and addressed requires a simple user 
> level program which anyone with a little, basic programming background can write today. Clearly the bar to creating these attacks you are pointing out is higher.

It's not a vulnerability unless the application lets it be one. It's a
feature, and no, it shouldn't be disabled just because one
interpretation of what is happening is "DOS attack".

Is it a DOS attack to receive lots of connections at once? Is it a DOS
attack to recieve a connection that opens then doesn't send data for a
while? All of these are legitimate behaviors of TCP - as is having the
receiver stall the sender.

Joe

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: MURALI BASHYAM <[email-masked]>
Date: Mon, 29 Oct 2007 22:44:52 -0700 (PDT)

----- Original Message ----
From: Joe Touch <[email-masked]>
To: Mahesh Jethanandani <[email-masked]>
Cc: [email-masked]
Sent: Monday, October 29, 2007 3:42:41 PM
Subject: Re: [tcpm] Is this a problem?

Mahesh Jethanandani wrote:
> Folks,
> 
> We have documented a case of HTTP servers that are prone to resource
> starvation with the use of a small user level program. The program
 does
> not require any special privileges or changes in the kernel. The user
> level program on the client opens a connection to a HTTP server,
 sends a
> GET request for a large file (larger than the advertised window of
 the
> client) but never reads the response.
> 
> Three well-known, public sites were tested for this vulnerability. 
> The two most common HTTP servers, Apache and IIS were the target.
> While one site had put mitigation technique in place, the others had
> none. With the latter two we were able to hold connections in
> ESTABLISHED state for days. The former site had a mitigation in place
> with a fixed timeout of 11 min., which was easy to guess and work
> around.
> 
> We (the authors) believe that this is a huge problem. What do you
> folks feel?

and connections on all the TCP based applications out there? Which "application"

> Previous responses to this documentation has been that it is a
> application problem. It is clear from our experimentation that most
 HTTP
> servers (and FTP too) have not implemented any mitigation techniques.
 We
> believe that this problem exists across the whole range of TCP based
> applications prevalent on the internet, although our experiments were
> limited to the web application. Where applications have tried to put
> mitigation techniques in place, workaround has been easy. This is
 mainly
> because applications do not have the same amount of visibility as TCP
> does on the state of the connection.

Applications know when a connection hasn't closed. They also know when
 a
write would block due to insufficient socket resources. Either or both
of these provide sufficient visibility to avoid the problem entirely.

Applications can implement a fixed or even an adaptive timeout, but unless they can make
the above distinction i am pointing out, they will be knocking out legitimate (albeit slow) 
connections, i.e there will be a fairness issue. Applications do not have visibility into TCP connection
sender's state to solve the fairness issue. A solution which penalizes legitimate connections is clearly a broken one.

Joe

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com