Subject: [tcpm] New I-D

From: Mahesh Jethanandani <[email-masked]>
Date: Tue, 13 Feb 2007 10:24:19 -0800

A new I-D has been posted on the IETF web site.

http://www.ietf.org/internet-drafts/draft-mahesh-persist-timeout-00.txt
"TCP Maintenance and Minor Extensions", Mahesh Jethanandani, Murali Bashyam, 9-Feb-07, <draft-mahesh-persist-timeout-00.txt> 

Comments are welcome.
-- 

*Mahesh Jethanandani*
*Technical Leader*
**Corporate Development*
*
[email-masked] <mailto:[email-masked]>
Phone :*+1 408 527-8230*
Fax :*+1 408 527-6351*

**
170 West Tasman Drive
San Jose, CA 95070
United States
www.cisco.com <http://www.cisco.com>

This e-mail may contain confidential and privileged material for the 
sole use of the intended recipient. Any review, use, distribution or 
disclosure by others is strictly prohibited. If you are not the intended 
recipient (or authorized to receive for the recipient), please contact 
the sender by reply e-mail and delete all copies of this message.

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 13 Feb 2007 13:55:52 -0500

On Tue, Feb 13, 2007 at 10:24:19AM -0800, Mahesh Jethanandani wrote:
> "TCP Maintenance and Minor Extensions", Mahesh Jethanandani, Murali 
> Bashyam, 9-Feb-07, <draft-mahesh-persist-timeout-00.txt> 
> Comments are welcome.

It's good to see the description of the problem you observed along with
the solution, but it looks to me like this is more of an OS mechanism to
manage resources rather than a protocol-based mechanism.  Is this
correct?

-- 
Wesley M. Eddy
Verizon Federal Network Systems

---

From: David Malone <[email-masked]>
Date: Tue, 13 Feb 2007 20:17:47 +0000

> "TCP Maintenance and Minor Extensions", Mahesh Jethanandani, Murali Bashyam, 9

This idea seems quite closely related to TCP keepalives, though it
is addressing a different problem. Do you have any suggestions for
a sockets API to these persist timers?

The draft spends a lot of time showing that the application cannot
deal with the problem. I don't buy all the arguments in this section:
for example, an application can usually control the sockbuf sizes
and then either user non-blocking writes or timeouts to determine
if the application at the other end has gone away or not. I'm not
sure how often an application cares if the TCP at the other end is
still alive!

On the other hand, as pointed out, there are situations where there
is no longer any application (data has been handed off to TCP and
the application has exited/moved on). In these cases it could be
quite hard for an administrator to control the resource usage without
something like the timeouts that you describe. I know FreeBSD has
had to introduce some facilities for dealing with these sort of
resource issues (there is a per-process limit on how many sockbufs
can be allocated by a process and there is a "tcpdrop" utility to
force a TCP connection to be closed).

	David.

---

From: MURALI BASHYAM <[email-masked]>
Date: Tue, 13 Feb 2007 12:28:11 -0800 (PST)

It's a TCP protocol and connection state specific
solution to the infinite persist state issue. The
consequences of not hanndling this issue lead to a
resource starvation problem, along with other
problems, DDOS etc.

Murali

> manage resources rather than a protocol-based

____________________________________________________________________________________
Sucker-punch spam with award-winning protection. 
Try the free Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/features_spam.html

---

From: Fernando Gont <[email-masked]>
Date: Wed, 14 Feb 2007 05:01:31 -0300

At 03:24 p.m. 13/02/2007, you wrote:

Comments inline....

><http://www.ietf.org/internet-drafts/draft-mahesh-persist-timeout-00.txt>http://www.ietf.org/internet-drafts/draft-mahesh-persist-timeout-00.txt
>Bashyam, 9-Feb-07, <draft-mahesh-persist-timeout-00.txt> Comments are welcome.

What if I advertise a window of 1, instead?

Or, what if I advertise a window of zero, then before you abort the 
connection I advertise a window of a few bytes, and then I go back to 
advertising a window of zero (and so on)?

I think it is interesting to find a workaround for this type of 
resource exhaustion attack (as well as for Netkill, etc.).

However, I think the heuristics will need to be more complex. If not, 
it will be easy (and cheap) for the attacker to fool the proposed 
counter-measures. (the examples above are some possible ways to do so).

Kindest regards,

-- 
Fernando Gont
e-mail: [email-masked] || [email-masked]
PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1

---

From: MURALI BASHYAM <[email-masked]>
Date: Wed, 14 Feb 2007 00:14:17 -0800 (PST)

Good point.

This particular behaviour can be detected quite easily
when compared to a well-behaved receiver. A
well-behaved TCP receiver will do receive side silly
window avoidance, and would only advertise a window
increase when at least a MSS worth of buffer space is
available (typically). 

Murali

--- Fernando Gont <[email-masked]> wrote:

> >Bashyam, 9-Feb-07,
> <draft-mahesh-persist-timeout-00.txt> Comments are
> welcome.
> 

____________________________________________________________________________________
8:00? 8:25? 8:40? Find a flick in no time 
with the Yahoo! Search movie showtime shortcut.
http://tools.search.yahoo.com/shortcuts/#news

---

From: "Anantha Ramaiah \(ananth\)" <[email-masked]>
Date: Wed, 14 Feb 2007 00:35:59 -0800

Yep, on a related note, some TCP stacks shrink windows, window shrinking
is used as a convenience tool in some cases :-) We probably need to deal
with such cases as well, since the robustness principle allows it.

-Anantha 

> -----Original Message-----
> From: MURALI BASHYAM [mailto:[email-masked]] 
> Sent: Wednesday, February 14, 2007 12:14 AM
> To: Fernando Gont; Mahesh Jethanandani (mahesh)
> Cc: [email-masked]
> Subject: Re: [tcpm] New I-D (draft-mahesh-persist-timeout-00.txt)
> 
> Good point.
> 
> This particular behaviour can be detected quite easily when 
> compared to a well-behaved receiver. A well-behaved TCP 
> receiver will do receive side silly window avoidance, and 
> would only advertise a window increase when at least a MSS 
> worth of buffer space is available (typically). 
> 
> Murali
> 
> ><http://www.ietf.org/internet-drafts/draft-mahesh-persist-tim
> eout-00.tx
> >t>http://www.ietf.org/internet-drafts/draft-mahesh-persist-ti
> meout-00.t
> >xt
> > <draft-mahesh-persist-timeout-00.txt> Comments are welcome.
> > 
> > e-mail: [email-masked] || [email-masked] PGP Fingerprint: 7809 
> > 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
> > 
> > 
> > 
> > 
> > 
> 8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! 
> Search movie showtime shortcut.

---

From: Fernando Gont <[email-masked]>
Date: Wed, 14 Feb 2007 05:58:19 -0300

>with such cases as well, since the robustness principle allows it.

The amount of data queued for transmission, is another factor worth 
considering in the heuristics, as for DoS attacks there will usually 
be a large amount of queued data (send buffer full, or close to).

Also, if the corresponding socket has been close()ed by the 
application running at the TCP sender, the application will usually 
have no control on the socket anymore (unless the SO_LINGER (?) 
socket option was set?). This may be worth noting in the doc, too.

Kindest regards,

---

From: Mark Allman <[email-masked]>
Date: Tue, 20 Feb 2007 13:10:23 -0500

Regarding:

Wes:
> [...] but it looks to me like this is more of an OS mechanism to

I'm with Wes here.

I just took a spin through the draft and am left with three thoughts ...

  + You say apps don't have a good role to play even after citing apps
    that 'pause' as the culprit.  I can't make these things agree in my
    head.  Even in the absence of TCP-specific information it seems that
    these apps could time themselves out if no progress is being made
    (i.e. data getting exchanged).

    (Further, things like HTTP sends after which the app goes away and
    then getting wedged into persist don't worry me much.  That sounds
    like a once in a blue moon sort of situation to me.)

  + This seems fundamentally like an operating system resource issue to
    me and not something for a networking standards body.  OSes deal
    with all sorts of resource contention issues without standards.  Why
    does this problem need to be solved the same way by each OS?

  + In particular, I think you have solved the problem wrong.  Don't get
    me wrong ... maybe it is perfectly OK for your purposes and it is
    fine with me if you want to use it.  However, I would want something
    a little smarter to really avoid the mythical attacks you describe.
    All you did was to cap the length of connections.  So, conceivably a
    periodic attack could still keep all your resources busy.  It seems
    to me that you'd want to use some sort of scheme that actually took
    into account contention and resource exhaustion.  E.g., some sort of
    LRU what-has-been-in-persist longest when you need new resources and
    they are not available sort of culling scheme?  Or, when you have
    exhausted X% of your resources to keep some in reserve.  That might
    be wrong and busted itself (it is quickly off the top of my head),
    but it at least takes into account the resource contention unlike a
    naive timeout.

Just my two bits ...

allman

---

From: MURALI BASHYAM <[email-masked]>
Date: Tue, 20 Feb 2007 12:16:46 -0800 (PST)

>     (i.e. data getting exchanged).

Well behaved applications usually SHOULD do this, i
agree, can't be guaranteed though. Leaving the problem
to be addressed by the app, does not ensure timeliness
from the point of view that TCP resources such as
buffers and connections are scarce and by how much
etc. The scheme lacks robustness if left to the
application, and for it to be beneficial, ALL
applications running on a system should be doing it,
otherwise there is the weakest link.

>     does this problem need to be solved the same way

At first glance, it does seem like an OS resource
issue, but TCP connections can have data backlogged in
their send queues for a variety of reasons and for
variable amounts of time, i am not sure how a protocol
unaware OS based solution can make the distinction
between the network causing the backlog transiently,
and a legitimate vs rogue receiver causing the backlog
etc, i feel that making it TCP protocol specific
(something designed only for persist state) allows for
well controlled, granular solution(s) and heuristics
to detect specifically misbehaving receivers in this
scenario.

>     but it at least takes into account the resource

The LRU scheme can be overlaid on top of the timeout
scheme to ensure a flexible and adaptive solution
quite easily, i agree. It still calls for tracking
connections bottlenecked specifically by the infinite
persistence problem.

Murali

 
____________________________________________________________________________________
The fish are biting. 
Get more visitors on your site using Yahoo! Search Marketing.
http://searchmarketing.yahoo.com/arp/sponsoredsearch_v2.php

---

From: Mark Allman <[email-masked]>
Date: Tue, 20 Feb 2007 18:53:27 -0500

> otherwise there is the weakest link.

So, let me step back.  I should have been more careful.  I think this is
a problem with the draft-not the approach.  That is, I can certainly
understand why a TCP would want to not depend on apps.  But, the text in
the i-d doesn't lead me to that conclusion and I think it is at very
best dubious in some places.

> At first glance, it does seem like an OS resource issue, but TCP
> connections can have data backlogged in their send queues for a
> variety of reasons and for variable amounts of time, i am not sure how
> a protocol unaware OS based solution can make the distinction between
> the network causing the backlog transiently, and a legitimate vs rogue
> receiver causing the backlog etc, i feel that making it TCP protocol
> specific (something designed only for persist state) allows for well
> controlled, granular solution(s) and heuristics to detect specifically
> misbehaving receivers in this scenario.

We are having a terminology problem here.  I meant "OS" as in the thing
that contains the TCP implementation.  I did not mean that an OS could
blindly reap off resources without understanding anything about TCP.

Let me put it this way ... This seems to me to be a TCP
**implementation** issue.  Why should we need to standardize this issue
and not other internal resource contention issues?  Why is this anything
but a very local problem that can be handled by the TCP implementation
in whatever way that implementation chooses?

And, then why is the proposed solution the right one?

allman

---

From: MURALI BASHYAM <[email-masked]>
Date: Tue, 20 Feb 2007 16:26:49 -0800 (PST)

> best dubious in some places.

If it is dubious, then we can correct it, and perhaps
we need to explain the motivation for that more
clearly.

> issue, but TCP

Okay, we are on the same page :-).
> in whatever way that implementation chooses?

It is an implementation to correct a problem caused by
the protocol's REQUIREMENT for a sender to be in an
infinite persistence state, thus allowing for abuse by
a misbehaving receiver side application (or by a lot
of well-behaved receivers). The draft is outlining a
solution to the problem (there could be others). The
idea in the (informational) draft is to highlight to
the group that requirement of the protocol, its
consequences and a possible solution.

Are you suggesting that any requirements of the
protocol specification that lead to a potential denial
of service type of scenarios and thus causing internal
resource contention, and/or any other network issues
are all to be considered 'implementation' ? Shouldn't
we look at the requirements of the protocol a little
more carefully?

> And, then why is the proposed solution the right

It does solve the problem it sets out to based on a
"total time allowed to be in persist state" type of
cap on the duration.

Are you implying there could be other behaviours from
the receiver that may not be addressed here, such as
the receiver opening the window a little and then
closing it that fernando mentioned, thus leading to a
longer term issue? We need to build heuristics into
the solution that can detect this and other types of
anomalous receiver behaviour, and take appropriate
action. And the heuristics themselves need to be based
on a notion of a well behaved receiver in this
scenario, to avoid false positives.

Murali

 
____________________________________________________________________________________
TV dinner still cooling? 
Check out "Tonight's Picks" on Yahoo! TV.
http://tv.yahoo.com/

---

From: Fernando Gont <[email-masked]>
Date: Tue, 20 Feb 2007 21:41:08 -0300

>in whatever way that implementation chooses?

I think there are some instances of this already... e.g., Clark's RFC 
on fragment reassembly. IMHO, an Inforamtional doc on some 
implementation issues that may be troublesome might be of help.

>And, then why is the proposed solution the right one?

So far it does not look like "the right one" to me. I raised at least 
a few ways in which the attacker can fool the proposed countermeasure.

---

From: Mark Allman <[email-masked]>
Date: Wed, 21 Feb 2007 09:44:54 -0500

> we look at the requirements of the protocol a little

Two things:

  + I think it is a big stretch to say this requirement should hold in
    resource constrained situations.  There are a zillion examples where
    a system is supposed to do something, but if it can't do it then it
    can't do it.  And, if a system cannot handle all the established +
    requested connections then something has to give---and, what that is
    ought to be a policy decision.

  + If you do want to read the standard as requiring a TCP to keep this
    connection in the face of everything then you don't need an
    informational because you cannot change a protocol requirement with
    an informational.

> Are you implying there could be other behaviours from

I am not implying anything.  In my previous note I said that a simple
timeout is a poor solution to the problem because it does not actually
take into account that resources are constrained.

Let me try an example...  Let's say we set this timer to 10 minutes.  A
TCP can stay in persist for 10 minutes.  Two possibilities:

  + The 10 minutes goes by and this is the only connection on the
    machine. 

    Why purge the connection?  The ends are both actively still
    interested and the connection is not causing any problems.

    (Clearly, we could say this is just the machine's policy---that
    connections in persist for 10 minutes get purged.  And, that is fine
    if someone wants to do that.  But, in this case the connection is
    posing no problem and so purging the connection is not addressing a
    problem, just being done due to some sort of house keeping policy.)

  + The 10 minutes goes by and the machine is heavily loaded and has
    turned away dozens or hundreds of connections because of this
    overloaded situation.

    In this case, purging the connection may offer a little relief and
    let other connections come in.  Maybe these will complete and this
    is good.  Or, maybe this is an attack and so the next connection
    will just wedge us for another 10 minutes.

    The point here is that any relief we see from this purging is by
    happenstance and not because we did something to actually address
    the resource constraint problem.  Why did we wait 10 minutes?  Why
    not 9?  Or 11?  If we want to deal with resource issues then we need
    to kick something in when resources are limited.  Kicking some
    connections out at some arbitrary point is just a *hope* that in
    some cases it will help.

What am I missing here?

allman

---

From: MURALI BASHYAM <[email-masked]>
Date: Wed, 21 Feb 2007 11:13:02 -0800 (PST)

> protocol requirement with

I am not sure what the status of the draft is supposed
to be, would experimental be a more appropriate
status?
Aside from the status of the draft, the more
fundamental point is whether we agree that this is a
protocol specific requirement leading to a problem for
which there is a protocol specific/aware solution we
can put in place. 

>     connections out at some arbitrary point is just

I thought we agreed that a resource based demand
driven scheme (based on who has been idle for the
longest time) can be used on top of the timeout based
policy scheme. Doesn't that address the issue are
raising here?

Thx,
Murali

____________________________________________________________________________________
Do you Yahoo!?
Everyone is raving about the all-new Yahoo! Mail beta.
http://new.mail.yahoo.com

---

From: "Caitlin Bestler" <[email-masked]>
Date: Wed, 21 Feb 2007 12:47:20 -0800

MURALI BASHYAM wrote:

> 
> I am not sure what the status of the draft is supposed to be,
> would experimental be a more appropriate status?
> Aside from the status of the draft, the more fundamental
> point is whether we agree that this is a protocol specific
> requirement leading to a problem for which there is a
> protocol specific/aware solution we can put in place.
> 

I have always assumed the following:

	Any application MAY disconnect from any peer when
	it no longer finds maintaining the connection with
	the peer to be of value.

	An application SHOULD disconnect from any peer when
	continued support of that peer will have an adverse
	impact on other connections, especially when the
	peer is acting suspiciously.

Therefore any application MAY and perhaps SHOULD disconnect
from a suspiciously idle peer, especially before maintaining
the idle connection adversely impacts active connections.

Are you suggesting that there is anything in any protocol
requirement that contradicts this?

---

From: John Heffner <[email-masked]>
Date: Wed, 21 Feb 2007 17:37:47 -0500

I think I agree with Wes, David and Mark that this is more of an OS than 
a protocol issue (if I'm characterizing their comments correctly), and 
Fernando that the approach is not powerful enough to solve the broader 
problem when viewed as an attack.

To expand a bit, I had a hallway conversation a couple years ago with 
Stanislav Shalunov about his netkill [1] attack, and more generally 
about contention for send buffer memory.  (I think the problem you're 
addressing falls precisely into this broader category.)  What I took 
away from this (Stas may or may not agree) is that since the resource 
contention occurs at the operating system level (competing for scarce 
kernel memory), that is the right place to arbitrate the contention. 
Each host may want to apply a different policy based on what it 
considers important.

As proof of concept, I wrote up a simple script (I think only about 30 
lines of python) that used the TCP ESTATS MIB to monitor connections, 
and kill off ones making no progress while consuming the most memory. 
This proved an effective defense against netkill, and I think would 
solve the problem described in the draft as well.  It would be easy 
enough to write a kernel thread to do the same if the system doesn't do 
ESTATS.

   -John

[1] http://shlang.com/netkill/