Subject: Re: [tcpm] DoS attack from misbehaving receivers

From: Mark Allman <[email-masked]>
Date: Mon, 05 Feb 2007 09:24:25 -0500

I have thought abou this attack and had some spirited discussion with
Rob on the topic.  I am not greatly worried about it myself.

  + It is pretty easy to detect this attack (in fact there is a footnote
    in the paper that says the author's experiments were detected by
    their network provider!).  It is fairly easy to see that more data
    is being ACKed than is being actually received.

  + Slammer was a one packet UDP fire and forget situation.  It sourced
    traffic as fast as the attached link could support---no control of
    any kind.  (Other worms have had this property, as well.)  We did
    not see a melting core.  So, I am not quite sure I am worried about
    a botnet of optack-ing boxes coaxing something to happen that has
    not already happened.

  + If we think this is a problem that needs a solution, we should think
    about how to do so without hacking things like changing the sending
    order.  (E.g., a generalized nonce (a la Savage) or something.

My two bits ...

allman

_______________________________________________
tcpm mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/tcpm

---

From: Rob Sherwood <[email-masked]>
Date: Mon, 5 Feb 2007 16:41:41 -0500

Hi Mark,

>     is being ACKed than is being actually received.

If I recall correctly, the majority of the previous spirited discussions
surrounded this point :-)  To summarize my end of that discussion,
the attacker's ACK rate creates a trade-off between the detectability
of the attack versus the attack's amplification factor.  The paper focuses
on the theoretic maximum attack rate, i.e., sending one ACK per window,
without concern of detection.

I believe that an intelligent attacker can construct ACKs with reduced
amplification that exactly mimic the ACK stream of a valid connection,
making this attack impossible to detect in general given only the sender's
local information.  However, even with the amplification rate reduced to 
avoid detection, the amount of traffic generated is still significant.

For example from RFC2581: Sec 4.2, the attacker could generate ACKs for
every other segment, and space each ACK 100ms apart, in effect emulating
a slow connection (2 segments/100ms=~ 3000bytes/100ms=~ 29KB/s).  It is
not clear to me that there exists any method for the sender to locally,
passively differentiate this malicious connection from a completely
valid one because the ACK streams are identical.

While a single connection at this speed is uninteresting for DDoS, the
attacker could repeat this, in parallel, across many hosts, resulting
with a large aggregate amount of traffic, only limited by the attacker's
ability to produce ACKs.  Sending ACKs for every other segment produces
an amplification factor of 3000/40 or 75x the attacker's bandwidth,
implying that a single attacker on a 100Mb/s link (for example, at a
University) could in theory generate 7500Mb/s of traffic, which is more
than a OC-96 link (4976.64 Mb/s) and most of the capacity of an OC-192
link (9953.28 Mbit/s ).  So, this is a variant on the attack that is both
difficult to detect and has potential to be dangerous.

Further, the footnote is somewhat misleading.  The point of that
particular experiment was to maximize the amount of traffic induced from
a single host, not to avoid detection.  Had the experiment tried to avoid
detection, for example, by using the ACK rates described above, it is
highly unlikely the attack would have been detected.  Operators simply
cannot have alerts generated for every ~30KB/s stream on their network.

>     a botnet of optack-ing boxes coaxing something to happen that has

But the set of hosts susceptible to the optack attack (i.e., everything
that implements TCP) is orders of magnitude larger then the set of hosts
susceptible to the Slammer attack (i.e., everything that ran Microsoft's
SQL server).  I assume someone will correct me if I'm wrong, but I do not
believe we have seen a worm attack that caused even the majority of access
links to saturate, which is exactly what I contend optack+botnets can do.

>     order.  (E.g., a generalized nonce (a la Savage) or something.

I agree that the generalized nonce solution (particularly, the cumulative
nonce solution proposed by Savage et al.) is a more elegant solution.
However, it requires both sender and receiver code to change, limiting
its incremental deployability.  If the deployability issues could be
addressed, this is clearly the right way to go, IMHO.

- Rob
.

---

From: Mark Allman <[email-masked]>
Date: Mon, 05 Feb 2007 21:14:46 -0500

Rob-

> [...]
> making this attack impossible to detect in general given only the
> sender's local information.  However, even with the amplification rate
> reduced to avoid detection, the amount of traffic generated is still
> significant.

Maybe I was not clear enough above.  I think I agree with you that
detection on the sender-side is hard---although, a sender that is
continuously pounding is actually quite noisy and getting only ACKs that
acknowledge large amounts of sequence space is suggestive.

What I meant was that at the receiver side you can detect this attack
pretty readily.  If X bytes are received and 10X bytes are ACKed
(regardless of how many ACKs are used or how they are spaced) then
something is wrong.  

Basically, in terms of sequence space you cannot mimic the flow normal
flow of ACKs and get any amplification.  If you want to amplify then you
have to change things and those changes can be observed.

> particular experiment was to maximize the amount of traffic induced
> from a single host, not to avoid detection.  Had the experiment tried
> to avoid detection, for example, by using the ACK rates described
> above, it is highly unlikely the attack would have been detected.

I understand.  I found the footnote to be illustrative in that
conceivably nobody was looking for your 'attack' traffic and yet they
found it and took action anyway.  To me this says that if you want to
amplify and maximize things then people will find you.

> Operators simply cannot have alerts generated for every ~30KB/s stream
> on their network.

I agree.  However, I can't believe you are actually suggesting using
optacks to get 30KB/sec flows.  You don't need optack at that point.
Further, for some detection methods (i.e., more being ACKed than
received) you don't need an alert---you just need to send a RST to the
server and kill the connection because it isn't doing any useful work.

> SQL server).

Three things ...

  + Optack is a secondary attack.  That is, a host will only do
    optimistic ACKing once it has been compromised in some other
    fashion.  So, saying that Slammer is somehow a bad example because
    it is "small" is not quite right, IMO.

  + I understand that one compromised machine can optack against
    somewhat arbitrary servers.  (Although, note that you need to find
    places to slurp from that actually have a reasonable amount of data
    such that they could in fact impose a load on the network.)  But,
    you are still going to be somewhat limited by the number of
    compromised hosts you can bring to bear on the DDoS.

  + My point in bringing up Slammer is not that it is precisely the same
    as what you are suggesting.  My point is that this was a case where
    75K hosts were pounding the network as hard as they possibly could
    and the core did not melt and there was no congestion collapse.  I
    understand that you could theoretically bring more firepower to bear
    on the network, but I am not sure that in practical terms you could
    ever really do it.

> I agree that the generalized nonce solution (particularly, the
> cumulative nonce solution proposed by Savage et al.) is a more elegant
> solution.  However, it requires both sender and receiver code to
> change, limiting its incremental deployability.  If the deployability
> issues could be addressed, this is clearly the right way to go, IMHO.

What is the "deployability" issue?  That it cannot happen as quickly as
a sender-only hack?  This all comes down to (1) do we think this is a
problem that really needs to be addressed and, if so then (2) do we
think this needs to be addressed immediately or would a cleaner approach
that took a bit longer to engineer and deploy be fine?  I am not sure I
even think this is a huge problem we are facing and so I am not in favor
of short term hacks when we know how to do better.  I understand that we
disagree on this point (which is fine).

allman

---

From: Rob Sherwood <[email-masked]>
Date: Tue, 6 Feb 2007 18:19:52 -0500

> acknowledge large amounts of sequence space is suggestive.

My confusion - I thought you were referring to detecting this on the
sender's side.  I agree that detecting the attack on the receiving
end is more straight forward, but it's not clear to me that relying
on receiver-side network operators to solve this problem is always
practical/desirable.  While this is a more philosophical argument,
it is a well established one in the DDoS literature: if your network
is under attack, you want to solve the problem locally, not depend on
operators at the source of the attack to solve it for you.  I consider
the issues with IP spoofing and the lack of pervasive egress filtering
a reasonable data point of why source-side solutions are not practical.
However, if this group feels that source-based prevention techniques
are sufficient, then I guess I should stop bugging you :-)

> optacks to get 30KB/sec flows.  You don't need optack at that point.

I was suggesting that you need optack to get *many* 30KB/s flows, where
the total aggregate traffic is larger than possible without optack.  

>     compromised hosts you can bring to bear on the DDoS.

I think I've been unclear.  So if an attacker compromises K machines,
and uses them to optack attack N hosts, then you are saying that the
attack is limited because an attacker cannot make K big enough?  I think
I'm missing something, but my assumption was that this the attacker
would be launching this from a botnet, which there are studies that
show the existence of botnets on the order of 10K+ hosts and larger.
Having a K in the 10K range with any sort of amplification factor would
seem to be a significant threat to me.

Additionally, I'm assuming the reason this issue was brought back up (and
to this list) was that Michal Zalewski recently found a vulnerability in
deployed versions of Apache and IIS that allows for a malicious client
to request arbitrary data from a web server.  While this is not terribly
interesting in general, it means that every web server, not just those
with large files, are now potential optack targets.

http://www.securityfocus.com/archive/1/455833/100/0/threaded

>     on the network, but I am not sure that in practical terms you could

Well, the short story is that I'm not 100% certain that the attack
is practical either.  My current research involves trying to decide
the attack's practicality  without actually running it.  This includes
some network topology mapping to evaluate the Internet's vulnerability
to optack and non-optack bandwidth flooding attacks, and simulations to
ensure that the attack is not self-limiting.  However, I think it is safe
to say that even if it is not practical to cause collapse in the core,
there is still danger to access links such that it is still worth while
spending time discussing possible solutions.

> disagree on this point (which is fine).

So I have backed off of my original stance w.r.t (2) since our previous
conversation about this: the paper has been out for over a year now and
people are not exploiting this attack in the wild, so clearly there is
time to engineer a better solution if one is available.

That said, I maintain that any proper solution should only modify the
sender, where existing cleaner solutions modify both sender and receiver.
Because the change affects security, any New Solution should not be
optional or negotiated at connection time - both ends of the connection
need to support it, else the attack vector persists and the solution
accomplishes nothing.  It does no good to have a solution where the client
can just pretend to not support it.  If we require that all senders and
receivers implement New Solution, then it creates interoperability issues
with TCP implementations that do and do not yet support New Solution.
This is what I mean by the deployability issue.

- Rob
.

---

From: Mark Allman <[email-masked]>
Date: Tue, 06 Feb 2007 22:32:30 -0500

> sender's side.  I agree that detecting the attack on the receiving end
> is more straight forward, but it's not clear to me that relying on
> receiver-side network operators to solve this problem is always
> practical/desirable.  While this is a more philosophical argument, it
> is a well established one in the DDoS literature: if your network is
> under attack, you want to solve the problem locally, not depend on
> a reasonable data point of why source-side solutions are not
> practical.  However, if this group feels that source-based prevention
> techniques are sufficient, then I guess I should stop bugging you :-)

But, your line of thinking here is that it is the *network* that is
under attack, right?  I.e., the compromised machine (the sink) and the
machine it is coaxing into a higher-than-usual sending rate are merely
pawns and there may be no big effects on either of these hosts or their
networks, right?  So, I am not sure the experience you cite above is
germane.  

Also, in thinking about this more I think you could detect *something
weird* on the sender side of the attack (i.e., the side receiving the
optacks).  One could think in terms of the symmetry work that Kreibich,
et.al. presented at hotnets a couple years ago.  The sender-side would
certainly see a much larger ACK ratio than one ACK / two data packets,
right?  (If not then there is no amplification.)  This is not as good of
a test as the receiver-side test of ACKing more data than arrives in
that it is not iron clad.

> Additionally, I'm assuming the reason this issue was brought back up
> (and to this list) was that Michal Zalewski recently found a
> vulnerability in deployed versions of Apache and IIS that allows for a
> malicious client to request arbitrary data from a web server.  While
> this is not terribly interesting in general, it means that every web
> server, not just those with large files, are now potential optack
> targets.
> 

Oh, ugh.  (With or without optack.)

> Well, the short story is that I'm not 100% certain that the attack is
> practical either.  My current research involves trying to decide the
> attack's practicality without actually running it.  This includes some
> network topology mapping to evaluate the Internet's vulnerability to
> optack and non-optack bandwidth flooding attacks, and simulations to
> ensure that the attack is not self-limiting.  However, I think it is
> safe to say that even if it is not practical to cause collapse in the
> core, there is still danger to access links such that it is still
> worth while spending time discussing possible solutions.

I look forward to seeing this work.

> So I have backed off of my original stance w.r.t (2) since our
> previous conversation about this: the paper has been out for over a
> year now and people are not exploiting this attack in the wild, so
> clearly there is time to engineer a better solution if one is
> available.

(And, the attack was made public by others before your paper in an I-D.
Even presented at a TCPM meeting, I believe.  (Or, maybe tsvwg.))

> sender, where existing cleaner solutions modify both sender and
> receiver.  Because the change affects security, any New Solution
> should not be optional or negotiated at connection time - both ends of
> the connection need to support it, else the attack vector persists and
> the solution accomplishes nothing.  It does no good to have a solution
> where the client can just pretend to not support it.  If we require
> that all senders and receivers implement New Solution, then it creates
> interoperability issues with TCP implementations that do and do not
> yet support New Solution.  This is what I mean by the deployability
> issue.

I understand this take.  Certainly there is logic in here.  But, let me
come at it from a different angle.  Security is all bout policy.  So,
maybe big web servers that don't want to be attacked just rate limit
connections unless they use the new mechanism.  Say, they allow some
number of bytes to be sent without restriction, but if something big is
being transmitted (i.e., we're in the heavy tail) then some client is
not going to get good performance unless they turn on New Solution.  If
they don't then they will get suboptimal performance.  Of course, if
we're looking at things in this detail a policy could also be to limit
bandwidth when connections are sending lots of data, seeing no loss and
seeing big stretch ACKs and we don't need a mechanism at all.

The other thing that is still rolling around my head is that this is
claimed (rightly, IMO) as an attack on the *network*.  Further, you are
saying that a jillion tiny flows can make up the attack.  Fine.  So, it
is not clear to me why any end host has much incentive to do something
to help this problem much.  It is not going to be an issue at the edges
of the networks, only when things aggregate up.  Of course, then also
everyone has an incentive to keep the shared part of the network working
well---which would then argue that a scheme that involved both endpoints
might work out OK.

I am still not convinced that we need to mess with the protocols at all
here, however.

allman