Subject: Re: [dnsext] comments on draft-crocker-dnssec-algo-signal-03

From: Eric Osterweil <[email-masked]>
Date: Sat, 1 Aug 2009 11:05:27 +0200

On Jul 31, 2009, at 8:24 PM, Michael Graff wrote:

> On Jul 30, 2009, at 21:29, Paul Vixie <[email-masked]> wrote:
>>>
>> what would it take to convince you?
>
> I would like to see someonefind out two things.
>
> One, if the bottleneck and fear of something that gas been around  
> for many years (DNS over tcp)

... but with smaller packet sizes (i.e. those w/o crypto), this  
problem was not apparent.  Message sizes (apparently) did not  
exacerbate path limitations.  Advertising a 4096B buffer for a 600B  
response would not get you into trouble.  However, advertising a 4096B  
buffer for a response that could vary between 2K and 4K could bring us  
to where we are today.

> is on the equipment, os, or DNS server code.

The evidence has been presented in a few venues (including IETF 75).   
In addition to that presentation you can look at the more detailed  
material presented at RIPE 58:
	http://secspider.cs.ucla.edu/docs.html
or any of the other relevant published work listed on that page.  You  
can even download dnsfunnel and look for yourself
	http://www.vantage-points.org/download.html

I think, perhaps, the latter might be the best way to assess the  
problem, as the tool will allow you to see it for yourself.

>
>
> Two, how is this not a ddos vector waiting to happen today?
>
>
> That is, provisioning against stress and planning on ddos, how would  
> an admin react differently in server and network capacity planning?

It's no more or less of a DDoS vector than before, but now the servers  
are forced to deal w/ a higher sustained load (of legitimate traffic)  
than before.  A higher load that seems (based on the evidence) to  
include unnecessary TCP traffic in many cases.

>
>
> In short, why is the increase if tcp in real queries a real problem  
> when in a properly chosen algorithm and with reasonable ttl values,  
> it can be minimized?

This is not entirely clear.  What is a "properly chosen" algorithm  
when the resolver behavior varies based solely on _where the resolver_  
is querying from?  Consider what we are seeing: a name server serves  
keys and some resolvers can retrieve them over UDP w/o a problem.   
While, at the same time, others resort to TCP.  It isn't the name  
server that has done anything differently.  otoh, if a resolver can't  
get the message simply because it is prompting the name server to send  
messages with authority and additional data, and it could avoid this  
by specifying a smaller buffer size (not 512, but something more  
likely to work), then it seems hard to claim that this fault exists in  
either "equipment, [or] os."

All of that said, if I read your comment correctly, I do completely  
agree with you that one thing operators of name servers could try to  
do is avoid some of this by taking steps to minimize the sizes of the  
sets in its zone (when this is possible).  Smaller keys should fit  
better, but I don't think this frees the resolvers from the  
responsibility of being smarter.

Eric

---

From: Douglas Otis <[email-masked]>
Date: Mon, 03 Aug 2009 11:26:01 -0700

On 7/31/09 11:24 AM, Michael Graff wrote:

> I would like to see someone find out two things.
>
> One, if the bottleneck and fear of something that has been around for
> many years (DNS over TCP) is on the equipment, OS, or DNS server code.

UDP fragmentation can overwhelm reassembly resources.  Fallback to TCP 
is not as robust against attack compared to SCTP.  For example, SCTP 
allows initial state to be held in initialization cookies, and not by 
servers.  Persistent SCTP connections offer roughly the same performance 
obtained by UDP, and with better control over congestion, while 
encouraging use of IPv6.

> Two, how is this not a DDoS vector waiting to happen today?

It is a question of the resources needed to either defend against:

a) highly amplified UDP reflected attacks

b) TCP resource exhaustion

In either scenario, _substantial_ increases in resources is necessitated.

SCTP should reduce resource consumption, retain low latency, and protect 
from reflected and cache poisoning attacks.  SCTP could be adopted by a 
relatively few ISPs as an assured transport (a safe harbor) during 
periods of abuse to limit the level of disruption that attacks might 
cause.

> In short, why is the increase if TCP in real queries a real problem when
> in a properly chosen algorithm and with reasonable TTL values, it can be
> minimized?

DNS attacks can request non-existent resources.  Non-existent resource 
requests can represent a poisoning vector, while also necessitating 
unique responses not mitigated by TTL values.  When responses leads to 
TCP fallback, more resources are consumed, latency and amplification is 
increased.

-Doug

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/namedroppers/>

---

From: Anand Buddhdev <[email-masked]>
Date: Tue, 04 Aug 2009 15:57:01 +0200

On 31/7/09 20:24, Michael Graff wrote:

> One, if the bottleneck and fear of something that gas been around for
> many years (DNS over tcp) is on the equipment, os, or DNS server code.
> 
> That is, provisioning against stress and planning on ddos, how would an
> admin react differently in server and network capacity planning?
> 
> In short, why is the increase if tcp in real queries a real problem when
> in a properly chosen algorithm and with reasonable ttl values, it can be
> minimized?

Michael,

I'm speaking here as an operator of an anycast network of DNS servers. I
am concerned that TCP is not suited to anycast. Current TCP levels are
low enough that any problems, if they exist, are too small to be
noticeable. If more DNS traffic were to be transported over TCP, then
there would be more broken TCP connections caused by "POP switching".

DNS anycast has been a big success because both UDP and anycast are
stateless. Throw in TCP, and it's not so simple any more.

Regards,

Anand Buddhdev
DNS Services Manager, RIPE NCC

---

From: Bob Halley <[email-masked]>
Date: Tue, 4 Aug 2009 08:58:11 -0700

> I would like to see someonefind out two things.

Me too.

I also do not understand why there seems to be such concern about TCP use in
forgery resilience.

Nominum has used TCP as part of the forgery resilience strategy in our
caching nameserver products for some time now.  In our testing, and in
real-world usage statistics, TCP has been a completely insignificant
contributor to total DNS traffic.

(These are also very short-lived connections, and wouldn't be a cause for
anycast concern either.)

Also, if there were TCP-related DoS issues with part of the DNS
infrastructure, it seems like that would be a problem whether TCP was used
in forgery resilience or not.

/Bob

---

From: Paul Vixie <[email-masked]>
Date: Tue, 04 Aug 2009 17:33:31 +0000

> From: Bob Halley <[email-masked]>
> Date: Tue, 4 Aug 2009 08:58:11 -0700
> 
> I also do not understand why there seems to be such concern about TCP use
> in forgery resilience.
> 
> infrastructure, it seems like that would be a problem whether TCP was
> used in forgery resilience or not.

people don't ddos tcp/53 because nobody's depending on it, not because
any fool with a one line perl script couldn't knock it down and pin it.

a wise (younger) man once told me that if i had an idea that worked on a
whiteboard and in a lab, i should multiply all constants by six million
and see if things still looked good.

---

From: bert hubert <[email-masked]>
Date: Tue, 4 Aug 2009 19:51:53 +0200

> and see if things still looked good.

Wow, DNS over UDP with 16 bit id field seems really dumb that way!

    Bert

---

From: Paul Vixie <[email-masked]>
Date: Tue, 04 Aug 2009 18:21:31 +0000

> From: bert hubert <[email-masked]>
> Date: Tue, 4 Aug 2009 19:51:53 +0200
> 

oh yeah, very much so.  but recall that transport security was an
afterthought not only in DNS but in IP (see source address repudiability
and random IP ID) and TCP (see random initial sequence numbers) and SMTP
(see unwanted traffic).

in DNS the TXID was only meant to help align incoming responses with
outstanding requests, not to prevent spoofing.  in that narrow sense,
16 bits was fine, since 65535 outstanding requests ought to have been
enough for any requestor-responder pair.

---

From: Andreas Gustafsson <[email-masked]>
Date: Wed, 5 Aug 2009 10:29:18 +0300

> any fool with a one line perl script couldn't knock it down and pin it.

Perhaps not many depend on TCP now, but as DNSSEC gets more widely
deployed, more and more people will depend on it, especially when
resolvers fall back to advertizing small EDNS0 receive buffer sizes to
deal with networks that don't handle fragmented UDP.  If the TCP side
of the DNS infrastructure can in fact be trivially DoSed, that's a
real problem that needs to be solved.
-- 
Andreas Gustafsson, [email-masked]

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 5 Aug 2009 08:20:38 -0600

> real problem that needs to be solved.

Alternately, resolvers could avoid naively falling back to TCP when a  
more appropriate buffer size would work.  When TCP is needed (because  
messages just won't fit over the path between a resolver and a name  
server), that is the end of the story and TCP should be used.   
However, TCP shouldn't be overused just because resolvers don't try  
more sane values between 4096 and 512.

A miscreant can, of course, still bomb port 53 w/ TCP, but keeping  
legitimate queries on UDP where possible spares name servers  
unnecessary load.

Eric

---

From: Paul Vixie <[email-masked]>
Date: Wed, 05 Aug 2009 14:27:41 +0000

> Date: Wed, 5 Aug 2009 10:29:18 +0300
> From: Andreas Gustafsson <[email-masked]>
> 
> ... If the TCP side of the DNS infrastructure can in fact be trivially
> DoSed, ...

yes.

> ... that's a real problem that needs to be solved.

one of the reasons i've been ignoring the current round of discussion
about whether tcp is evil or not, is that i'd rather work on a proposal.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 05 Aug 2009 14:57:50 +0000

> From: Eric Osterweil <[email-masked]>
> Date: Wed, 5 Aug 2009 08:20:38 -0600
> 
> ... However, TCP shouldn't be overused just because resolvers don't try  

is this input for EDNS0-bis, specifically, new fallback advice?  if so then
i hope it's been overtaken by events.  the approach i recommend is, try EDNS0
and if it doesn't work, give the hell up, go back to DNS, forget about DNSSEC
and any other EDNS-enabled feature for that transaction and for other
transactions to that server for some period of time, after which, retry EDNS0.

rationale: when there is something in the middle of the network that corrupts
packets, the endpoints should detect this but not correct it.  correcting it
should be left to the operators and implementors of the middleboxes.  users
whose service level is lower because of the corruption, should complain.

> legitimate queries on UDP where possible spares name servers unnecessary
> load.

a security-aware design requires that we not paint the target on our ass in
the first place.  if TCP/53 is trivially ddos-able, then we have to expect
such ddos's to take place the moment there is value in launching them.  so,
our own correct but unnecessary load is not a worthy consideration, only our
dependence and/or modalities.

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 5 Aug 2009 08:34:02 -0700

> whose service level is lower because of the corruption, should  

I think this advice should be different these days.  The in-path  
corruption for recursive resolvers is commonly an inability to handle  
fragments, and for end-hosts its most commonly an inability to handle  
fragments or an assumption that DNS=512B.  Actually not being able to  
handle EDNS0 at all is the minority case from the observations we  
reported.

Thus the fallback should probably be:
EDNS0 @ 4096
EDNS0 @ 1400
EDNS0 @ 512
with each using a fallback to TCP on truncation, and only if EDNS0 @  
512B doesn't work should one decide "No EDNS, give up"

And it is, IMO, impossible to get the operators and middleboxes to  
change.  Look at the pulling-teeth hastle people have on trying to  
get, say, 2-wire to fix trivial vulnerabilities in their NATs, let  
alone more subtle denial of service conditions.

Thus the algorithm/fallback should understand the common causes of  
corruption and attempt to work around them, out of operational  
necessity.

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 5 Aug 2009 09:56:39 -0600

On Aug 5, 2009, at 9:34 AM, Nicholas Weaver wrote:

> corruption for recursive resolvers is commonly an inability to  
> handle fragments, and for end-hosts its most commonly an inability  
> to handle fragments or an assumption that DNS=512B.  Actually not  
> being able to handle EDNS0 at all is the minority case from the  
> observations we reported.
>

I would completely agree with this.  Especially since there is a lot  
of evidence that resolvers often CAN overcome these issues by  
adjusting their buffer sizes.  It's a fix that can go into one of the  
components that can be directly addressed; the resolver.  That is, a  
resolver operator who wants to fix this problem can upgrade their  
resolver w/o addressing middleboxes that she may or may not be  
responsible for.

Eric

---

From: Douglas Otis <[email-masked]>
Date: Wed, 05 Aug 2009 11:46:14 -0700

On 8/5/09 8:56 AM, Eric Osterweil wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>> change. Look at the pulling-teeth hastle people have on trying to get,
>> say, 2-wire to fix trivial vulnerabilities in their NATs, let alone
>> more subtle denial of service conditions.
>>
>> corruption and attempt to work around them, out of operational necessity.
>
> I would completely agree with this. Especially since there is a lot of
> evidence that resolvers often CAN overcome these issues by adjusting
> their buffer sizes. It's a fix that can go into one of the components
> that can be directly addressed; the resolver. That is, a resolver
> operator who wants to fix this problem can upgrade their resolver w/o
> addressing middleboxes that she may or may not be responsible for.

Middle boxes will experience about 1.5 times the traffic that EDNS0@4096 
alone might have created.  This traffic might already be at an 
amplification of 80.  The 1.5x increased traffic and 3x increased 
latency will aggravate packet loss during ongoing attacks and may cause 
higher levels of TCP fallback during attack scenarios.  With TCP 
fallback, amplifications increased to 4 times that of a functional 
ENDS0@4096.

Now consider the use of SMTP authorization macros embedded within cached 
resource records that can further increase the number of transactions of 
often large TXT records by factors that range from 10 to 100, without 
costing the attacker additional resources beyond what was used to spam. 
  The macros allow local-parts to modulate the requests.  The macro 
processing has become embedded within several MTAs and various MUA spam 
filters.

-Doug

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 5 Aug 2009 11:54:28 -0700

On Aug 5, 2009, at 11:46 AM, Douglas Otis wrote:
>> of
> Middle boxes will experience about 1.5 times the traffic that  
> EDNS0@4096 alone might have created.  This traffic might already be  
> at an amplification of 80.  The 1.5x increased traffic and 3x  
> increased latency will aggravate packet loss during ongoing attacks  
> and may cause higher levels of TCP fallback during attack  
> scenarios.  With TCP fallback, amplifications increased to 4 times  
> that of a functional ENDS0@4096.

There is nothing to say that such fallback shouldn't be stateful: Once  
you know what an authority's level of response is, the results should  
obviously be cached, which eliminates the extra RTTs.

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 5 Aug 2009 13:43:06 -0600

On Aug 5, 2009, at 12:54 PM, Nicholas Weaver wrote:

> There is nothing to say that such fallback shouldn't be stateful:  
> Once you know what an authority's level of response is, the results  
> should obviously be cached, which eliminates the extra RTTs.

Indeed, resolvers already maintain per-nameserver information such as  
RTT.

Eric

---

From: Paul Vixie <[email-masked]>
Date: Wed, 05 Aug 2009 20:31:01 +0000

i really should be writing a specific proposal rather than debating this;
my apologies for my lack of self discipline as shown in two replies below.

---

> From: Nicholas Weaver <[email-masked]>
> Date: Wed, 5 Aug 2009 08:34:02 -0700
> 
> > rationale: when there is something in the middle of the network that
> > corrupts packets, the endpoints should detect this but not correct it.
> > correcting it should be left to the operators and implementors of the
> > middleboxes.  users whose service level is lower because of the
> > corruption, should complain.
> 
> fragments or an assumption that DNS=512B.  ...

unless we intend that all dns protocol agents contain this level of 
complexity, and that all future large-UDP WAN applications (like EDNS)
contain this level of complexity, and that both DNS and other large-UDP
WAN applications continue to ratchet up this body count in the indefinite
future, then we have to let the middle solve itself.

> ... Thus the fallback should probably be:
> EDNS0 @ 4096
> EDNS0 @ 1400
> EDNS0 @ 512
> with each using a fallback to TCP on truncation, and only if EDNS0 @  512B
> doesn't work should one decide "No EDNS, give up"

given that each fallback can involve a timeout, the aggregate global
application and end-user state load implied by having that many fallbacks
is way too high.  DNS is supposed to be quick and lightweight even when it
fails.

> And it is, IMO, impossible to get the operators and middleboxes to change.

of course not.  we're going to have to do DNS over XML over TCP/445 (HTTPS)
when a user or their DNS protocol agent knows that or detects that it is
behind a corrupting middlebox (or inside of a corrupting ISP).  we already
know this, and we know that no amount of EDNS or fragmentation fixage will
avoid the need for it.

---

> Date: Wed, 5 Aug 2009 09:56:39 -0600
> 

this target is still moving and will keep moving.  any complexity we add to
the endpoints to deal with corrupting middleboxes will simply re-baseline
the market for even more-corruptive middleboxes in that possible future.  i
don't plan to play whack-a-mole with ISP's and NAT vendors for the rest of
eternity and i don't think anybody else should do so either.

we're experiencing information warfare here.  people in the middle want their
revenue and/or their control, and they will impinge/corrupt our packets as
necessary to get the win they need.  they do not want reliable end-to-end
DNS to be possible, or they don't care whether end-to-end DNS can evolve new
features.  either way they own the middle and their goals are in opposition
and if we want to deliver our information we have to stop assuming that the
problems we're solving are accidental and/or sitting still.

---
===

---

From: Douglas Otis <[email-masked]>
Date: Wed, 05 Aug 2009 14:35:33 -0700

> obviously be cached, which eliminates the extra RTTs.

The concern has been from behavior in the form of distributed attacks. 
Botnets lend themselves to this type of activity.  Millions of DNS 
servers might be making requests against uncached resources from 
infrequently seen servers.  Do not assume attackers will confine 
themselves to a set of known servers, or that even completing a first 
pass with exponential back-off occurs before initial requests time out.

-Doug

---

From: Eric Osterweil <[email-masked]>
Date: Wed, 5 Aug 2009 21:08:18 -0600

> future, then we have to let the middle solve itself.

The proposal here is to keep the intelligence at the ends.  Much like  
TCP adjusts its window size at the ends, these agents can adjust their  
buffer sizes at the ends.

> is way too high.  DNS is supposed to be quick and lightweight even  

Well, some of these fallbacks result in immediate TC bits being set.   
Intelligent guessing and persistent state (with periodic reassessment)  
would allow for occasional timeouts, but not consistent timeouts.  For  
example, if 4096 times out, a quick 512 can tell the resolver if the  
name server is online (not followed by a TCP connection).  Then a more  
intelligent UDP-search of the space (perhaps starting at 1400)  can  
proceed.  Remember, getting TC bits is better than getting timeouts,  
so a search of the space should keep that in mind.

>> change.

This sounds like throwing the baby out with the bath water.  Yes,  
sometimes messages will not fit over the path.  No, this is not always  
the case.  UDP can, and should, continue to be the best option for DNS  
queries and responses, and TCP (or TCP/445...) can be a fallback, but  
there doesn't seem to be a good reason overreact.

> eternity and i don't think anybody else should do so either.

This is about making the components of the system more robust.  You  
can't say that since there are middlebox problems resolvers should  
avoid trying to overcome surmountable issues.  This is a very simple  
adjustment that uses all of the mechanisms that exist now.  By your  
own pen, EDNS0 allows resolvers to address this obstacle through  
clueful choice of buffer sizes, and if done well, this can be done  
with less overhead than a TCP connection.

Prognosticating that vendors will sense weakness and pounce on us with  
increasingly stupid appliances seems... An odd conclusion.  However,  
hamstringing our own systems to send them a message seems counter  
productive.

> problems we're solving are accidental and/or sitting still.

"Never ascribe to malice that which is adequately explained by  
incompetence" I really don't know that this is a matter of warfare.  I  
don't know what business model is benefited by dropping DNSKEY  
messages at this early stage of the deployment.  If there is an on- 
path attacker (say an ISP) then there are serious troubles no matter  
how you slice it.  However, without such an attacker, we still see  
PMTU problems, and many of them can be solved with a little bit of  
logic by using the provisions that already exist today (as described  
in a few of the above emails).

Seriously, I think there is a very simple set of adjustments that can  
be made that, in many cases, will alleviate some of the frivolous TCP  
connections.

Eric

---

From: Douglas Otis <[email-masked]>
Date: Thu, 06 Aug 2009 07:46:49 -0700

>> my apologies for my lack of self discipline as shown in two replies
[]
> The proposal here is to keep the intelligence at the ends. Much like TCP
> adjusts its window size at the ends, these agents can adjust their
[]
> proceed. Remember, getting TC bits is better than getting timeouts, so a
> search of the space should keep that in mind.

With hundreds of millions of botnet systems that might offer DNS, 
expansion of server state info must guard against overflow.  During 
overflow of the acquired state, resulting traffic should be considered. 
DNSSEC already increases network amplifications, where the added 
fallback attempts will aggravates this problem by a factor of 2 to 4. 
SCTP causes less traffic and will better deals with congestion.

>> know this, and we know that no amount of EDNS or fragmentation fixage
>> will avoid the need for it.

HTTP or HTTPS will require a dramatic increase in needed resources and 
will inhibit simple conversions.  Even so, TCP will not achieve 
comparable resilience.  SCTP offers better protection without modifying 
DNS messages.  Rather than expensive HTTP or HTTPS services, SCTP to UDP 
appliances could convert DNS messages for those that do not wish to 
otherwise alter their infrastructure.  Those that don't want to adopt 
SCTP in some fashion, might need to forgo DNSSEC.

> there doesn't seem to be a good reason overreact.

Would reflected attacks or DDoSing of TCP offer a reason?

-Doug

---

From: Paul Vixie <[email-masked]>
Date: Thu, 06 Aug 2009 15:24:12 +0000

> Date: Thu, 06 Aug 2009 07:46:49 -0700
> From: Douglas Otis <[email-masked]>
> 
> HTTP or HTTPS will require a dramatic increase in needed resources and will
> inhibit simple conversions.  Even so, TCP will not achieve comparable
> resilience.  SCTP offers better protection without modifying DNS messages.

if folks want to do uncorrupted DNS inside corrupting ISPs, today's choice
is use a VPN.  perhaps that's a good enough choice but i don't think it
supports opendns well and i know i get a little irritable when i have to
tunnel my DNS over SSH.  i think an XML profile for DNS, carried over HTTPS,
would be useful and popular for stub-to-recursive traffic.  provisioning at
the server side is an economics question for the I.T. department (would they
rather support VPN's?) and for opendns (how can they deliver their services
to comcast's customers?) and so while i know HTTPS has a very high cost i
still expect it to be a lower cost than existing alternatives for some folks.

> Rather than expensive HTTP or HTTPS services, SCTP to UDP appliances could
> convert DNS messages for those that do not wish to otherwise alter their
> infrastructure.  Those that don't want to adopt SCTP in some fashion, might
> need to forgo DNSSEC.

SCTP seems way more useful in recursive-to-authority flows.  florian's note
on nanog today (which i'll fwd here in its entirety) highlighted the short
lifespan of the usual stub resolver context.  someone else on nanog pointed
out that NSCD (or presumably LWRESD) could hide the shortness of that life
time, but i don't think we want to force people to run local forwarders.

---

From: Mark Andrews <[email-masked]>
Date: Fri, 07 Aug 2009 10:27:45 +1000

Years ago (well over a decade) I created a local DNS proxy listening
on 127.0.0.1 that just tracked response times from the nameservers
listed in resolv.conf and forwarded the queries based on that using
its own id space with retries.  The stub resolver used connected
sockets to talk to it and fell back to the nameservers listed in
resolv.conf if it was not available.  The idea was to handle dead
nameservers in resolv.conf more efficently.  Such a proxy could
keep the SCTP state for all the stub resolvers in a machine.  It's
just a little more shared state.

The proxy would need to manage EDNS buffer sizes these days in
addition to the id space.  The proxy could even sign requests on
behalf of the stub resolvers allowing DH to be used efficiently.

The proxy would not be used if the stub resolver re-set the
nameservers.

This also helps with long running applications as only the proxy
needs to track nameserver changes in resolv.conf.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Paul Wouters <[email-masked]>
Date: Sun, 9 Aug 2009 16:05:23 -0400 (EDT)

> needs to track nameserver changes in resolv.conf.

Uhm, are you going to add validation to your "stub" next?

Paul