Subject: [dnsext] Distributing the root zone out of band

From: Matthew Dempsky <[email-masked]>
Date: Fri, 5 Jun 2009 10:53:15 -0700

On Fri, Jun 5, 2009 at 10:11 AM, Olafur Gudmundsson <[email-masked]> wrote:
> Bill,
> you are a root server operator,
> what order of magnitude of concurrent TCP sessions are YOUR DNS servers
> prepared to handle?
>         10^x  x = ?

If we're worried about overloading the root servers, why not advocate
more caches download the root zone out of band?  E.g., it's already
possible to download a zlib compressed root zone file from
http://www.internic.net/domain/ along with a PGP signature (signed by
the same people that will hold the root zone's DNSSEC keys), verify
it, and configure your DNS cache to use it.  This can be easily
automated by a monthly cron job.

Large TLD operators can then stop worrying about making sure their
delegation records fit into a single 512 byte packet.

It's also possible to distribute the root zone file and signature
using mechanisms like BitTorrent to further distribute load.

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/namedroppers/>

---

From: Paul Vixie <[email-masked]>
Date: Fri, 05 Jun 2009 21:41:44 +0000

> Date: Fri, 5 Jun 2009 10:53:15 -0700
> From: Matthew Dempsky <[email-masked]>
> 
> If we're worried about overloading the root servers,

we're not.  it's any authoritative server.  the root servers are being
used as a narrow example but the com, com.br, co.uk, cn, and other zones
are going to have the same issue.

> why not advocate more caches download the root zone out of band?

because it goes stale, and gets deliberately amended, leading to chaos.

---

From: [email-masked]
Date: Fri, 5 Jun 2009 21:47:53 +0000

> are going to have the same issue.

	hence the ammended query... "concurrent TCP sessions to nameservers"

> because it goes stale, and gets deliberately amended, leading to chaos.

	there are plans to significantly change the dynamic on/in
	the root zone.  designing a stratagy that presumes a fairly
	small, static root zone might be short sighted.

--bill

---

From: Paul Vixie <[email-masked]>
Date: Fri, 05 Jun 2009 22:22:58 +0000

> Date: Fri, 5 Jun 2009 21:47:53 +0000
> From: [email-masked]
> 

whether talking merely about concurrency or not, tcp/53 doesn't scale to
within four orders of magnitude of the QPS of udp/53.

---

From: [email-masked]
Date: Fri, 5 Jun 2009 23:08:02 +0000

On Fri, Jun 05, 2009 at 02:57:42PM -0700, Matthew Dempsky wrote:
> On Fri, Jun 5, 2009 at 2:47 PM,  <[email-masked]> wrote:
> Plans to significantly change the size and/or dynamic nature of the
> root zone might be short sighted if the future requires a small,
> [slowly changing] root zone.

	yes.  but it seems clear that the POB have a very different view
	on what the future requirement might be.  if you have thoughts on
	this...

	http://www.icann.org/en/public-comment/public-comment-200907.html#root-scaling

--bill

---

From: [email-masked]
Date: Fri, 5 Jun 2009 23:09:53 +0000

> within four orders of magnitude of the QPS of udp/53.

	sure. and???  Olafurs question still stands.

--bill

---

From: Paul Vixie <[email-masked]>
Date: Fri, 05 Jun 2009 23:27:01 +0000

> Date: Fri, 5 Jun 2009 23:09:53 +0000
...
> 	sure. and???  Olafurs question still stands.

the answer is, the number is so low as to not be worth measuring precisely.
any solution that ends in "or else we'll fall back to tcp/53" is quite doomed.

---

From: David Conrad <[email-masked]>
Date: Fri, 5 Jun 2009 16:48:36 -0700

>> why not advocate more caches download the root zone out of band?

Great idea!

> because it goes stale, and gets deliberately amended, leading to  

Not if the root zone is slaved off a (set of) masters.  Like, oh say,  
root.iana.org...

Regards,
-drc

---

From: Paul Vixie <[email-masked]>
Date: Sat, 06 Jun 2009 00:48:13 +0000

> From: David Conrad <[email-masked]>
> Date: Fri, 5 Jun 2009 16:48:36 -0700
> 

does iana want the rootops to start pulling the root zone from that server?

---

From: Matthew Dempsky <[email-masked]>
Date: Fri, 5 Jun 2009 18:23:30 -0700

> does iana want the rootops to start pulling the root zone from that server?

No.  Open source software distributors have figured out how to
efficiently mirror files much, much larger than the root zone.  The
root zone operators should follow their model.

Example: According to [1], the Ubuntu Linux distribution has 300
mirrors with total bandwidth exceeding 250 Gbps available over HTTP,
FTP, and RSYNC.  According to [2], mirroring a single Ubuntu release
requires at least 30GB, and a full mirror takes at least 200GB.

I think the Internet has the resources to mirror the dozen or so files
at http://www.internic.net/domain/. :)

[1] https://launchpad.net/ubuntu/+archivemirrors
[2] http://www.ubuntu.com/getubuntu/mirror/2

---

From: [email-masked]
Date: Sat, 6 Jun 2009 01:58:13 +0000

> any solution that ends in "or else we'll fall back to tcp/53" is quite doomed.

	there you go Olafur, Paul has answered your question for me. :)

--bill

---

From: Paul Vixie <[email-masked]>
Date: Sat, 06 Jun 2009 02:09:43 +0000

> Date: Fri, 5 Jun 2009 18:23:30 -0700
> No.

are you the iana?

> Open source software distributors have figured out how to efficiently
> mirror files much, much larger than the root zone.  The root zone
> operators should follow their model.

since i was doing this at gatekeeper.dec.com in 1989 and my boss was doing
it at ftp.uu.net even earlier, and since my day job now hosts or operates
mirrors for just about every f/l/oss project, i am amused by your lecture.

> at http://www.internic.net/domain/. :)

changing them every day, plus more often in emergencies, and avoiding the
ever present tendancy to let them get out of date or concatenate local TLDs
into each host's mix, are the problems.  not the bits.

but we are far afield.  the root zone isn't the interesting part.  nor the
other files on www.internic.net.  it's all authority zones, everywhere, and
i don't think we want to rsync those to every endsystem.  (we abandoned the
old HOSTS.TXT file for several reasons, of which "scale" was one.)

---

From: David Conrad <[email-masked]>
Date: Fri, 5 Jun 2009 19:19:40 -0700

> does iana want the rootops to start pulling the root zone from that  

Well, I'm told IANA doesn't exist, so it would be difficult for it to  
want anything.  I'm also told the root servers were independent and  
thus, even if IANA were to exist, the root servers operators would do  
what they thought best for the communities they believe they serve  
regardless of what anyone else wants.

:-)  (if anyone actually needed this)

More seriously, root.iana.org is part of a two testbeds (the other  
being ns.iana.org) ICANN set up for folks to test a DNSSEC-signed  
root.  I expect they'll both be discontinued when the root gets  
signed.  However, this is somewhat orthogonal to what Matthew was  
proposing.  The point is that your concern about staleness of the  
'cached' root zone would not be an issue if people slave off a (set  
of) servers set up to respond to root zone transfer requests.

Regards,
-drc

---

From: Jim Reid <[email-masked]>
Date: Sat, 6 Jun 2009 07:07:48 +0100

On 6 Jun 2009, at 00:48, David Conrad wrote:

> Not if the root zone is slaved off a (set of) masters.  Like, oh  
> say, root.iana.org...

How many TCP queries can this set of masters handle? :-)

---

From: Florian Weimer <[email-masked]>
Date: Sat, 06 Jun 2009 09:01:10 +0200

> into each host's mix, are the problems.  not the bits.

We need this for the trust anchor(s) anyway (including non-DNS
fallback for the download).  Once we've got that, it's not a
significant challenge to ship more than just DNSKEYs---except that the
root is already fragmented and we don't know what contents our users
see or want, so this is unlikely to work on a global scale.  (I don't
know what this means for the trust anchors, but I think we'll know the
answer in a couple of months.)

---

From: David Conrad <[email-masked]>
Date: Sun, 7 Jun 2009 14:38:07 -0700

> How many TCP queries can this set of masters handle? :-)

Given the data is relatively stable (and thus could be replicated in  
an Akamai-like scheme) and the response requirements for zone transfer  
aren't anything like what they are for normal TCP queries, I'm not  
worried that the machines behind something like root.iana.org wouldn't  
be able to handle pretty much any load thrown at them.  Also, since  
the data is signed, it wouldn't really matter where/how you get it...

Regards,
-drc

---

From: Florian Weimer <[email-masked]>
Date: Mon, 08 Jun 2009 08:22:05 +0200

> the data is signed, it wouldn't really matter where/how you get it...

And if it's a zone transfer, you don't even leak traffic data to
strangers if you get it from the wrong place.

---

From: [email-masked]
Date: Mon, 8 Jun 2009 11:10:14 +0000

> strangers if you get it from the wrong place.

	er.. TCP has its own warts.

	I'm intriged by the implcit challange tossed by David.

	"... not worried that the machines behind something like 
	root.iana.org wouldn't be able to handle pretty much any 
	load thrown at them." - drc 

	this harks back to Olafurs question to me that was ably answered
	by Paul retorts that TCP is "epic fail" as the prefered transport
	for DNS queries.

	the brief thought experiment.

	10,000 QPS, each done over TCP.  Thats the three-way handshake.
	assume the normal 5% drop of the handshake  due to transient link 
	failure and a 2 minute timeout before the host drops the halfopen TCP
	connect.

	root.iana.org will need to sustain that 10,000 TCP connects per second
	and keep track of the sustained 60,000 half-opens ... I'd -love-
	to look at that stack'o'hardware.

--bill (who is going to get reamed for his swag arithmatic)

---

From: Shane Kerr <[email-masked]>
Date: Mon, 08 Jun 2009 14:31:08 +0200

Bill,

> --bill (who is going to get reamed for his swag arithmatic)

Indeed you are.

Existing web servers can handle more than twice that load today:

http://www.litespeedtech.com/web-server-performance-comparison-litespeed-2.1-vs.html

Heck, *Apache* can handle 10,000 queries/second on a single box.

While this is probably 25% of what BIND 9 could do on a similar box, and
maybe 15-20% of what NSD could do, it's not too shabby.

This test was in a clean environment, so it is possible that your
concerns about packet drop are a real problem, but you'd want at least 2
boxes in a cluster in any real-world setup, so unless you performance
was dropped in half you'd still meet your (arbitrary) metrics.

Of course, in DNS the client closes the TCP connection, which is
suckful, and makes life miserable in general. I suppose it is too late
to change the protocol, but if we do add a SCTP transport then this is
something to consider.

--
Shane

---

From: [email-masked]
Date: Mon, 8 Jun 2009 13:02:35 +0000

> maybe 15-20% of what NSD could do, it's not too shabby.

	er... http queries over what transport Shane?

http://www.litespeedtech.com/docs/webserver/config/tuning/?hilite=tcp,performance, 

sez:

Max Connections	Go to top
Description: Specifies the maximum concurrent connections that the server can accept. It includes both plain TCP connections and SSL connections. It should not exceed the hard limit set by the server: 300 for Standard Edition. Once this limit is reached, the server will close Keep-Alive connections when they complete active requests. 

...

that 300 number sticks out like a sore thumb.  just a tad shy of the swag target of 10,000
TCP setups per second adn the ongoing tracking for the 60,000 halfopen (keep-alives) projected.

what am i missing here?

---

From: bert hubert <[email-masked]>
Date: Mon, 8 Jun 2009 15:24:04 +0200

> TCP setups per second adn the ongoing tracking for the 60,000 halfopen (keep-alives) projected.

The main thing that is the issue with DNS over TCP is that 1035 says
you 'should'  keep the connection open as a server for at least two
minutes.

This quickly translates into hundreds of thousands of open connections
under even moderate load.

FWIW, SCTP appears to exhibit this same behaviour automatically even
when using the API in datagram mode (keeping the 'association' alive).
It is unclear to me if SCTP indeed has a lower server-level footprint
than TCP.

FWIW2, PowerDNS has been closing TCP sessions after 2 seconds of idle
for the past 10 years and no one has complained so far. But if you
want to be up to the letter compliant, two minutes is what you
'should' do.

   Bert

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 08 Jun 2009 22:38:18 +0900

> what am i missing here?

*ANYCAST*, of course.

FYI, a server with multiple CPU chips or cores may distribute load
by distributing requests over chips or cores based on hash values
computed from source addresses, which is, in a sense, anycast.

						Masataka Ohta

---

From: bert hubert <[email-masked]>
Date: Mon, 8 Jun 2009 15:21:33 +0200

Bert

---

From: Florian Weimer <[email-masked]>
Date: Mon, 08 Jun 2009 15:49:24 +0200

> that 300 number sticks out like a sore thumb.

It's probably some restricted version of the server code.

> just a tad shy of the swag target of 10,000 TCP setups per second
> adn the ongoing tracking for the 60,000 halfopen (keep-alives)
> projected.

Tens of thousands of pending connection attempts per process haven't
been a problem for a while.

Keeping those TCP send buffers can be a challenge, but hacks such as
TransmitFile/sendfile are certainly possible.

I wouldn't be surprised if there were some large systems which can
serve more DNS transactions over TCP than over UDP because the UDP
path has a global lock in it.  TCP tends to be way more optimized than
UDP (or SCTP).  On the other hand, UDP is such as a low overhead that
this does not matter on UP or small SMP machines.

-- 
Florian Weimer                <[email-masked]>
BFK edv-consulting GmbH       http://www.bfk.de/
Kriegsstraße 100              tel: +49-721-96201-1
D-76133 Karlsruhe             fax: +49-721-96201-99

---

From: [email-masked]
Date: Mon, 8 Jun 2009 13:53:27 +0000

> under even moderate load.

	i was presuming 10,000 qps w/ a 5% drop == a sustained 60,000 halfopen
	connections plus the other 9,500 serviced connections - per second.
	the 60,000 is the 5% over the 2 minute interval.... not hundreds of
	thousands but enough... :)

> It is unclear to me if SCTP indeed has a lower server-level footprint

	the ns simulator seems to indicate that STCP has a somewhat higher
	foot print.. 

> want to be up to the letter compliant, two minutes is what you

	thats still a boatload'o'open TCP to keep track of.

--bill

---

From: [email-masked]
Date: Mon, 8 Jun 2009 13:58:59 +0000

> It's probably some restricted version of the server code.

	yeah ... its what they call the "standard release"

> TransmitFile/sendfile are certainly possible.

	indeed - but now we step into the wonderful world of 
	OS and stack tuning.  

> this does not matter on UP or small SMP machines.

	i would be (pleasently) surprised.

> Kriegsstra_e 100              tel: +49-721-96201-1

---

From: [email-masked]
Date: Mon, 8 Jun 2009 13:56:38 +0000

> computed from source addresses, which is, in a sense, anycast.

	ok... presume 300 (per lightspeed) and my swag of ~70,000 TCP connections
	to track per sec.   lets anycast the solution and we get...
	about 240 machines just to handle the current load that a single machine can
	do over UDP.   we'll avoid doign the power/cooling cost multiple for now.

--bill

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 07:13:58 -0700

BIll,

On Jun 8, 2009, at 4:10 AM, [email-masked] wrote:
> 	10,000 QPS, each done over TCP.

How frequently does the root zone need to be refreshed?

Regards,
-drc

---

From: Matt Larson <[email-masked]>
Date: Mon, 8 Jun 2009 10:27:35 -0400

On Sun, 07 Jun 2009, David Conrad wrote:
> Also, since the data is signed, it wouldn't really matter where/how
> you get it...

Please recall that the delegation NS RRSets and glue aren't signed and
these compose most of the root zone's contents.

So while we all know that this doesn't prevent successful validation
if a child zone is signed, it does mean that the entire contents of
the root zone cannot be validated even after being DNSSEC-signed.

Matt

---

From: Niall O'Reilly <[email-masked]>
Date: Mon, 08 Jun 2009 15:35:41 +0100

> How frequently does the root zone need to be refreshed?

	Currently,
	or in a future where new TLDs are added very frequently?

	Niall O'Reilly
	University College Dublin IT Services

---

From: Jim Reid <[email-masked]>
Date: Mon, 8 Jun 2009 15:46:30 +0100

> How frequently does the root zone need to be refreshed?

How many new TLDs is ICANN going to allow?

If each TLD changes its delegation once a year -- a reasonable  
estimate -- we're already looking at a new zone file for . every day  
or theresabouts with the current root.  If the number of TLDs go up,  
the frequency of updates (and root zone file versions?) will go up  
proportionately.

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 07:49:42 -0700

On Jun 8, 2009, at 6:56 AM, [email-masked] wrote:
> 	to track per sec.

Um.  This has all been quite amusing, but I don't recall anyone  
actually suggesting something like root.iana.org handling all DNS  
queries over TCP.

The suggestion was that caching servers have a copy of the root zone.   
Paul expressed worry that if this were to occur, the zone would get  
out of sync.  I countered by suggesting that could be addressed by  
slaving off a facility like root.iana.org (whether using {A,I}XFR is,  
of course, irrelevant).

Now, with that in mind, where do you get your "swag of ~70,000 TCP  
connections to track per sec"?

A different thought experiment:

According to http://dns.measurement-factory.com/surveys/200810.html,  
they estimate there are 12 million authDNS servers and were able to  
detect 4 million open resolvers.  Just for fun, lets combine (I know  
auth won't care, but this is a thought experiment) and round up and  
for the sake of argument say there are 20 million name servers that  
would need to fetch the zone and they will all do this via AXFR.  Now,  
the root zone is (generally) updated twice a day.  So, 20,000,000 /  
( 12 * 60 * 60 ) = 462 zone transfers per second. Another point to  
note: the data is not particularly dynamic, thus it could be trivially  
replicated via existing content delivery networks.

Are you worried that smart folks can't come up with an infrastructure  
that would support ~500 'file transfers' per second?

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 07:52:03 -0700

> 	or in a future where new TLDs are added very frequently?

New root zone entries do not change the schedule of when a new root  
zone is generated.  I would be surprised if this policy changed with  
the addition of new TLDs.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 07:54:07 -0700

> How many new TLDs is ICANN going to allow?

Irrelevant to the question asked.

> or theresabouts with the current root.

Currently, the root zone gets regenerated 2 times per day regardless  
of whether there are any changes.  I don't suspect this will change.

>  If the number of TLDs go up, the frequency of updates (and root  
> zone file versions?) will go up proportionately.

Nope.  It means the number of changes in the root zone per generation  
goes up.  It doesn't change the frequency of generation.

Regards,
-drc

---

From: [email-masked]
Date: Mon, 8 Jun 2009 16:01:01 +0100

> replicated via existing content delivery networks.

AuthDNS servers don't need the root zone - indeed many argue that 
authoritative servers *shouldn't* have the root zone at all.

Recursive DNS servers need it.

Ray

-- 
Ray Bellis, MA(Oxon) MIET
Senior Researcher in Advanced Projects, Nominet
e: [email-masked], t: +44 1865 332211

---

From: [email-masked]
Date: Mon, 8 Jun 2009 15:01:53 +0000

> How frequently does the root zone need to be refreshed?

	what do the ICANN SO's think?

--bill

---

From: Shane Kerr <[email-masked]>
Date: Mon, 08 Jun 2009 17:13:31 +0200

> How frequently does the root zone need to be refreshed?

I had the same question. At 10,000/second, every human on the planet can
download a new personal copy weekly.

--
Shane

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Mon, 8 Jun 2009 08:32:33 -0700

> 	do over UDP.   we'll avoid doign the power/cooling cost multiple  

I don't know what lightspeed is, but for modern http servers (nginx,  
varnish, ...) 30k requests per second is well within reasonable.

HAProxy which doesn't serve files but parse the requests and pass it  
on to another service can do more.   They have a test that only sets  
up the tcp session and they got > 100k/sec: http://haproxy.1wt.eu/10g.html

  - ask

-- 
http://develooper.com/ - http://askask.com/

---

From: Paul Vixie <[email-masked]>
Date: Mon, 08 Jun 2009 15:38:07 +0000

> From: Shane Kerr <[email-masked]>
> Date: Mon, 08 Jun 2009 14:31:08 +0200
> ...
> Of course, in DNS the client closes the TCP connection, which is suckful,
> and makes life miserable in general. I suppose it is too late to change
> the protocol, but if we do add a SCTP transport then this is something to
> consider.

yes, in an sctp transport, either end should be able to end an association.

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 08:44:15 -0700

Ray,

> Recursive DNS servers need it.

I know.  As I said:

"Just for fun, lets combine (I know auth won't care, but this is a  
thought experiment) and round up ..."

Alternative suggestions for more accurate numbers more than welcome.

Regards,
-drc

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 08:51:13 -0700

> 	what do the ICANN SO's think?

My guess is that they don't, at least about something like that.  As  
far as I'm aware, folks are happy with 2 x day.

Regards,
-drc

---

From: [email-masked]
Date: Mon, 8 Jun 2009 15:59:52 +0000

> actually suggesting something like root.iana.org handling all DNS  

	sorry, I suspect that I mis-read the following email:

Cc: Paul Vixie <[email-masked]>, Matthew Dempsky <[email-masked]>,
   [email-masked]
From: David Conrad <[email-masked]>
To: Jim Reid <[email-masked]>
In-Reply-To: <[email-masked]>
Subject: [dnsext] Re: TCP queries to the root servers
Date: Sun, 7 Jun 2009 14:38:07 -0700
X-Mailer: Apple Mail (2.935.3)
Precedence: bulk
List-ID: <namedroppers.ops.ietf.org>

> Jim Reid wrote:

"...I'm not
worried that the machines behind something like root.iana.org wouldn't
be able to handle pretty much any load thrown at them."

Regards,
-drc

	-----------------------------------------------
> connections to track per sec"?

	10,000 qps over UDP is the rough number reported by the four RSOs I've talked
	with this week. 

	empirical evidence points to about 5% of the TCP attempts are left 
	"half-open" - likely due to transport churn.  RFC 1034 says the TCP
	timeout to close those half-open connections is two minutes.

	so... moving the 10,000 qps to TCP, with 5% being left open -every second-
	will give me, after the first two minutes, a sustained "load" of 
	60,000 half-open TCP connections to manage ... and every second, I pick up
	another 500 and close 500.  leaving about 9,500 active TCP sessions per
	second to manage in addition to those other 60,000.  

	so to a first approximation, to do -todays- load over TCP, you'll need
	to handle about 70,000 TCP sessions per second.

	was that helpful?

> replicated via existing content delivery networks.

	ah... we see that you have stayed with the "AXFR" track, while
	Jim, myself, and others, have trotted down the track of move
	all DNS queries to TCP, not just AXFRs.

> that would support ~500 'file transfers' per second?

	no, not worried abt that, even dumb folks like me can
	do that.

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 09:13:04 -0700

> 	sorry, I suspect that I mis-read the following email:

My apologies -- I suspect my use of the term 'queries' led to the  
confusion.

Regards,
-drc

---

From: [email-masked]
Date: Mon, 8 Jun 2009 16:15:36 +0000

> varnish, ...) 30k requests per second is well within reasonable.

	lightspeed wa a site quoted by Shane and then me.
	I have no idea who they are either.

> up the tcp session and they got > 100k/sec: http://haproxy.1wt.eu/10g.html

	nifty thing..  using specialized HW(nics), tuned kernels, and
	even then, the test methodology claims:

	"...The request generator continuously connects to HAProxy to fetch 
	the selected object from the server in loops for 1 minute, with 
	500 to 1000 concurrent connections."   

	er... thats 1000 per minute or 17 concurrent TCP connections per second.
	buzt.  lets try again.

	I don't care about raw data throughput, i care about the number
	of concurrent TCP connections.  

--bill

---

From: Paul Vixie <[email-masked]>
Date: Mon, 08 Jun 2009 17:01:20 +0000

why are we talking about this in the exclusive context of the root zone?
the underlying protocol problems are shared by every authority zone, and
unless we're planning to mirror every authority zone in every recursive
server, we need to keep the protocol working without such mirroring.

re:

> Subject: Re: [dnsext] Re: TCP queries to the root servers
> To: David Conrad <[email-masked]>
> Cc: [email-masked]
> Organization: ISC
> Date: Mon, 08 Jun 2009 17:13:31 +0200
> X-Mailer: Evolution 2.26.1 
> Sender: [email-masked]
>

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Mon, 8 Jun 2009 10:25:03 -0700

> 	er... thats 1000 per minute or 17 concurrent TCP connections per  

No, that's not what it says -- but we're getting way off-topic.

All I wanted to point out is that the state of the art of TCP based  
servers (for HTTP) supports request rates high enough to the vast  
majority of DNS servers.

Keep using anycast[1] and the numbers really aren't that crazy.

  - ask

[1] http://www.nanog.org/meetings/nanog37/presentations/matt.levine.pdf

---

From: Paul Vixie <[email-masked]>
Date: Mon, 08 Jun 2009 17:42:32 +0000

> Date: Mon, 8 Jun 2009 07:52:03 -0700
> 
> New root zone entries do not change the schedule of when a new root zone
> is generated.  I would be surprised if this policy changed with the
> addition of new TLDs.

under some proposals we'd see millions of top level names, and in that case
i'd expect the rate of urgent NS or DS changes to be several-times-daily.  so
it would not surprise me if the policy had to change in that case.  in the
more likely case of a few dozen new top level names per year, i agree that
the root zone publication schedule would not have to change at all.

---

From: Paul Vixie <[email-masked]>
Date: Mon, 08 Jun 2009 18:07:02 +0000

i'm limiting this reply to just the issues related to tcp/53.  i've already
asked, several times, that we not treat the root zone as special, so we must
be prepared to do anything for COM and CO.UK that we would do to fix problems
we see in accessing the root zone.  (which rules out mirroring in my view.)

> From: Florian Weimer <[email-masked]>
> Date: Mon, 08 Jun 2009 15:49:24 +0200
> 

whatever number of simultaneous open tcp sockets you allow, even if
millions, can be exceeded due to attacks, failures, or flash crowds.

if the target initiated close after every response then i could imagine
being able to manage the state load, but as soon as one allows any kind
of timeout window, whether 30 seconds as in RFC 1035 or 2 seconds as in
powerdns, then you're painting a target on your back.

for SCTP i'd recommend that associations be left open and that it be
explicitly recommended and expected that responses be reordered.  that
way an initiator could dump all kinds of queries into the queue and the
target could answer them in any order; this would avoid any perceived
need to open several associations in parallel.

(i am guessing that TCP/53 initiators open sessions in parallel in this
case and/or expect/demand responses to be in request order though the RFC's
say otherwise, and that responders will generally respond in request
order.  an implementation survey should probably be done to find out.)

> I wouldn't be surprised if there were some large systems which can serve
> more DNS transactions over TCP than over UDP because the UDP path has a
> global lock in it.  TCP tends to be way more optimized than UDP (or
> SCTP).  On the other hand, UDP is such as a low overhead that this does
> not matter on UP or small SMP machines.

the state load of half-open connections is manageable due to syn flood
concerns first expressed a decade or more ago.  but the pool of fully open
connections is a fat target, even if one does not follow RFC 1035's
recommendation for server side connection management.

---

From: David Conrad <[email-masked]>
Date: Mon, 8 Jun 2009 14:29:20 -0700

Paul,

> why are we talking about this in the exclusive context of the root  

Because the root of the DNS is special (quantitatively if not  
qualitatively)?

> server, we need to keep the protocol working without such mirroring.

Everybody cares about the root.  If you screw up the root zone, it has  
a bit more impact than if you screw up (say) dv.isc.org.  If the  
authority for dv.isc.org is DDoS'd out of existence, it has a bit less  
impact than if the root were DDoS'd, etc.

I'm told a number of larger ISPs already mirror the root zone.  All  
I'm suggesting is to make this easier/more formalized for folks who  
want to do it.

Regards,
-drc

---

From: Paul Vixie <[email-masked]>
Date: Tue, 09 Jun 2009 03:11:45 +0000

> Date: Mon, 8 Jun 2009 14:29:20 -0700
> 
> I'm told a number of larger ISPs already mirror the root zone.  All I'm
> suggesting is to make this easier/more formalized for folks who want to
> do it.

i'm told a number of the ISP's who mirror the root zone also amend it.  i
will become more comfortable with this after the real one has been signed
so that such amendments can be filtered out.

in my hotel in beijing last week, all traffic toward any root name server
address was answered inside chinanet by chinanet's own servers.  not a
comfortable environment in which to consider wide scale root zone mirroring.

---

From: bert hubert <[email-masked]>
Date: Tue, 9 Jun 2009 09:47:34 +0200

[ Moderators note: Post was moderated, either because it was posted by
   a non-subscriber, or because it was over 20K.  
   With the massive amount of spam, it is easy to miss and therefore 
   delete relevant posts by non-subscribers. 
   Please fix your subscription addresses. ]

> comfortable environment in which to consider wide scale root zone mirroring.

With a hostile service provider, all bets are off. The best thing one
can do in that case is make sure you don't get *any* answers.  So it
does not make a lot of sense to ponder this situation for protocol
design, or even operational guidelines.

    Bert

---

From: Olafur Gudmundsson <[email-masked]>
Date: Tue, 09 Jun 2009 10:49:49 -0400

>comfortable environment in which to consider wide scale root zone mirroring.

I take this as an endorsement that the root zone be signed using NSEC
to make any such changes transparent.

         Olafur

---

From: Paul Vixie <[email-masked]>
Date: Tue, 09 Jun 2009 15:17:41 +0000

> From: bert hubert <[email-masked]>
> Date: Tue, 9 Jun 2009 09:47:34 +0200
> 

of course.  however, making local root name service universal, is going
to engender a lot more opportunistic hostility along these lines.  since
there's no operational problem being solved by it, and since the failure
modes include "becoming out of date" in a dozen ways, and since there
is almost nothing one can say the root zone needs that many 2LD's and
some 3LD's also need, this whole thread is "twilight zone" territory.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 9 Jun 2009 12:08:54 -0400

No hat.

On Tue, Jun 09, 2009 at 03:17:41PM +0000, Paul Vixie wrote:

> some 3LD's also need

I have to agree with Paul that the discussion of distributing the root
zone widely doesn't seem to be solving the basic problems that we
started the thread with.

As near as I can tell, the start of this topic was that things would
be better if we switched to TCP, which was immediately rejected as
entailing too much state on the authority servers.  That argument led
us to the possibility of mirroring the root zone more widely.

Since, as Paul points out, the problem is at least as bad for the
authority servers for large TLDs as it is for the root, we have either
established that the solution doesn't fit the problem, or else that we
need to distribute many zones so widely.  In the absence of a proposal
to ensure such wide mirroring that is not subject to all of the
problems that the HOSTS file had, I think we can conclude that the
proposal is not an answer to the problem of TCP state on authority
servers.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: David Conrad <[email-masked]>
Date: Tue, 9 Jun 2009 09:08:31 -0700

On Jun 9, 2009, at 8:17 AM, Paul Vixie wrote:
> this whole thread is "twilight zone" territory.

I agree, but for different reasons that I won't bother going into.

Back to the original point/question that started this thread:

Certain name servers are setting DO=1 but advertising a buffer size of  
less than 1220 as part of a heuristic to try to get around broken  
firewalls/NATs/etc. handling of EDNS0.  My reading of RFC 3226 would  
indicate this is a violation of the RFC.

The implication of this particular violation is that it is highly  
likely the root servers will see increased TCP fallback as name  
servers will set TC on the DNSSEC-related RR filled response  
immediately after the root is signed.  I believe there are two options  
to deal with this:

1) fix the name server implementations so they do not violate RFC 3226  
by not advertising a buffer less than 1220 when DO=1

2) revise RFC 3226 to remove/revise the buffer size restriction.

Does anyone other than Mark Andrews think we should choose option 2?

Regards,
-drc

---

From: Jim Reid <[email-masked]>
Date: Tue, 9 Jun 2009 17:56:53 +0100

On 9 Jun 2009, at 17:08, David Conrad wrote:

> Certain name servers are setting DO=1 but advertising a buffer size  
> of less than 1220 as part of a heuristic to try to get around broken  
> immediately after the root is signed.  I believe there are two  
> options to deal with this:
>
> 1) fix the name server implementations so they do not violate RFC  
> 3226 by not advertising a buffer less than 1220 when DO=1
>

I vote for both. Sort of. 3226 needs to be updated anyway because it  
talks about legacy stuff like RFC2525 compliant servers and resolvers.  
And A6 records. Though if 3226 does get revisited, I see no reason to  
change what it says about the minimum buffer size. Maybe if 2761bis is  
revived this would be a better home for defining appropriate minimum  
buffer sizes when the DO bit is set?

---

From: Paul Vixie <[email-masked]>
Date: Tue, 09 Jun 2009 17:18:17 +0000

> Date: Tue, 9 Jun 2009 09:08:31 -0700
> 

my reading gives the same result.

> The implication of this particular violation is that it is highly likely
> the root servers will see increased TCP fallback as name servers will set
> TC on the DNSSEC-related RR filled response immediately after the root is
> signed.  I believe there are two options to deal with this:
> 
> 1) fix the name server implementations so they do not violate RFC 3226 by
> not advertising a buffer less than 1220 when DO=1
> 

not i.  however, if we're going to choose option 2, then i'll suggest an
additional action, which is an RFC stating that DO=1 should be ignored if
the buffer size is less than 1220.  in fact, RFC 3226 can be read that way.

---

From: [email-masked]
Date: Tue, 9 Jun 2009 17:31:37 +0000

> Does anyone other than Mark Andrews think we should choose option 2?

	so the choice is:

	pick 1, and trigger hardware/firmware upgrades near the edges of the
	Internet and residual, persistent edge failure in the DNS for decades
	to come.

	pick 2, and kowtow to shortsighted hardware/firmware vendors and force
	a protracted decade-long migration in the changes to DNS transport
	protocols.

	I pick #1.

--bill

---

From: Paul Vixie <[email-masked]>
Date: Tue, 09 Jun 2009 17:33:06 +0000

> Date: Tue, 09 Jun 2009 10:49:49 -0400
> From: Olafur Gudmundsson <[email-masked]>
> 
> I take this as an endorsement that the root zone be signed using NSEC to
> make any such changes transparent.

yes, please.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 9 Jun 2009 11:20:04 -0700

On Jun 9, 2009, at 9:08 AM, Andrew Sullivan wrote:

> entailing too much state on the authority servers.  That argument  
> led us to the possibility of mirroring the root zone more widely.
>
> authority servers for large TLDs as it is for the root, we have  
> either established that the solution doesn't fit the problem, or  
> else that we need to distribute many zones so widely.  In the  
> absence of a proposal to ensure such wide mirroring that is not  
> subject to all of the problems that the HOSTS file had, I think we  
> can conclude that the proposal is not an answer to the problem of  
> TCP state on authority servers.

Agreed.  Is there interest in creating a draft that describes how SCTP  
fallback for DNS might be implemented?

 From our company's perspective, defending TCP services through  
massive distribution represents a sizable portion of overall corporate  
expenditures.  In the long view, it would be remiss not to explore  
more robust solutions that avoid critical infrastructure being  
replicated by orders of magnitude.

While SCTP is likely to require greater resources than what is  
required for UDP, SCTP offers faster connection setup than TCP.  For  
DNS, when compared to TCP, this represents an opportunity to offer  
data ahead of requests.  This additional information might enable  
distribution strategies, for example.

SCTP can permit DNS message alignment within packets, where individual  
messages can be acknowledged separately.  Header alignment eliminates  
message buffering and head of queue blocking that occurs with TCP.   
Once an association is established, SCTP operates in a manner that  
closely approximates UDP.

Unlike TCP, SCTP does not commit resources until a cookie is  
returned.  This property enables important defensive strategies.   
Source identification offers immunity from flood attacks by allowing  
abusive sources to be properly handled without an inadvertent denial  
of services.   Source spoofing is a costly problem for either UDP and  
TCP, that SCTP resolves effectively within the protocol.

Estimating resources required by SCTP should consider optimized  
handling of SCTP associations.  There is no reason to assume DNS will  
only use SCTP in a generic manner.

Of course, SCTP offers better transaction security and would prevent  
Kaminsky related attacks with or without DNSSEC.  As a minor note,  
SCTP error detection offers support for jumbo frames and provides  
better data integrity for unsigned portions of DNS messages.  The  
ability of either UDP or TCP checksums to detect common bus related  
errors is astonishing poor.

-Doug

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 9 Jun 2009 14:28:01 -0400

> fallback for DNS might be implemented?

I believe Paul Vixie already expressed interest in doing this.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: bert hubert <[email-masked]>
Date: Tue, 9 Jun 2009 21:15:51 +0200

> I believe Paul Vixie already expressed interest in doing this.

I did some tests, and added SCTP serving support to the PowerDNS
Recursor, and to our regression testing query tool ('sdig'). I have to
admit to not being an SCTP expert, the last paragraph of this message
contains some open questions.

To keep it brief, the summary is that SCTP uses at least twice as many
packets as UDP, and far more if the association is not kept open for a
long time. Furthermore, an open association consumes server resources,
and probably not less than TCP (even while sharing a file descriptor
between queries). The initial cookie does not mitigate this - every
association carries state that has to be carried around.

>From this, my tentative conclusion is that SCTP transport per-se does
not buy us anything over TCP - even though it offers a 'datagram-like'
coding experience.

The whole story:

Interestingly enough, from a coding perspective, setting up the
connection is almost entirely TCP like, whereas responding to actual
queries runs 100% correctly using the unmodified UDP codepath
(recvfrom, sendto). This is with the SCTP API in 'one-to-many' mode.

In my measurements, I've seen that at least when used out of the box,
SCTP uses 4 packets for each query/response exchange when an
association has already been setup (query, response, query-sack,
response-sack).

Association setup is 4 packets, teardown is 3. Two packets of the
setup can carry both the 'cookie' and the initial request, plus the
cookie ack and the first data ack, saving 2 packets. This reduces the
initial setup costs in terms of packets to 2.

This makes for a grand total of 9 packets (!) for one DNS query
carried by SCTP  that is not quickly followed by another.

Two queries in quick succession run to 13 packets, 3 queries squeeze
into 17 packets.

In my testing, I often encountered so called 'heartbeat' packets,
which further increase the packet count.

An open question is if all these 'SACK' packets can be turned off, or
if SCTP could offer the equivalent of 'NOLINGER' so we could save
another two packets (RST).

But as stated above, even if both answers are yes, we are looking at
very elevated packet counts, at no less cost than TCP (to the
operating system).

      Bert

---

From: Matthew Dempsky <[email-masked]>
Date: Tue, 9 Jun 2009 13:23:13 -0700

> association carries state that has to be carried around.

I don't have any SCTP experience, but these results seem discouraging
for DNS-over-SCTP adoption.  I'd be interested in knowing if these
results contradict any DNS-over-SCTP proponents'
experiences/expectations.

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 10 Jun 2009 06:23:11 +0900

[email-masked] wrote:

> Description: Specifies the maximum concurrent connections that
> the server can accept. It includes both plain TCP connections
> and SSL connections. It should not exceed the hard limit set
> by the server: 300 for Standard Edition.

> TCP setups per second adn the ongoing tracking for the 60,000 halfopen (keep-alives) projected.

I reconsidered the problem and noticed that the figure of 300
should comes from SSL computation and disk access to retrieve
various HTML files, none of which are applicable to root servers
just returning the same data stored in memory in plain text.

Keeping 60,000 connections halfopen for 120 seconds is not a
problem because, in average, only 500 of them needs timeout
processing and others are resting in memory.

Given that a pcb is 176B long, the amount of memory to keep
60,000 pcbs is about 10MB. These days, even secondary cache
can hold it.

It is inappropriate to mix figures from a though experiment
by Shane and from real world HTTP servers by Bill.

						Masataka Ohta

PS

As I briefly checked, to have 240 cores, you only need 3 small
(6U) blade servers, each of which has 10 blades, each of which
have 8 cores.

---

From: Masataka Ohta <[email-masked]>
Date: Wed, 10 Jun 2009 06:30:55 +0900

> to make any such changes transparent.

An obvious solution is to solve the problem end to end by using
a resolver at your home (or home ISP).

I actually did so several times when resolvers of ISPs local
to my laptop were malfunctioning.

						Masataka Ohta

---

From: Paul Vixie <[email-masked]>
Date: Tue, 09 Jun 2009 23:16:54 +0000

> From: Douglas Otis <[email-masked]>
> Date: Tue, 9 Jun 2009 11:20:04 -0700
> 

i've taken a strong interest in this, but not as a fallback.

---

From: Paul Vixie <[email-masked]>
Date: Tue, 09 Jun 2009 23:36:23 +0000

> Date: Tue, 9 Jun 2009 13:23:13 -0700
> I don't have any SCTP experience, but these results seem discouraging for
> DNS-over-SCTP adoption.  I'd be interested in knowing if these results
> contradict any DNS-over-SCTP proponents' experiences/expectations.

in SCTP it takes three packets to set up an association (which is like a
TCP session only lightweight and spoofproof.)  it takes four packets to
shut down an association.  so, setup and shutdown are expensive compared
to UDP or TCP, and it only makes sense to use SCTP if you're able to keep
the association open and use it for multiple transactions.

it takes two packets to carry a request and a response, and if they are
the last request and response in the queue then each of these will be
followed by an ACK packet.  so, a slow 1TPS flow would take four packets
per transaction, whereas a faster flow would take two packets per
transaction.  the extra ACK-only packets don't delay the DNS result, but
they do appear on the wire.

by lightweight i mean that a root or TLD or 2LD/3LD server could have many
hundredthousands or even millions of associations open, all on a single
socket and therefore a single file descriptor, so it's not a burden on
the DNS server's system call interface (think of a select() mask here, or
a poll() filedes array.)

the two-byte length field from TCP/53 need not be carried in SCTP, since
the transport will tell the receiver the length of each incoming message.
messages can be quite long, say 64KByte.  the transport is reliable, so if
some segment (packet) is dropped in transit it will be retransmitted and
the receiver won't hear about it until the full message has landed.  there
is no IP fragmentation (any more than there is in TCP, and for the same
reason).

SCTP associations are spoofproof in a way that lets us go back to just
setting DNS TID to 0 at the start of each association, and incrementing it
by one each time we send a new transaction.  there's no need for PING and
neither a reason nor a method for randomizing source port numbers.

most importantly, there's no trivial way to force a downgrade, so if an
initiator tries SCTP first and only uses UDP when SCTP fails, then it will
only use UDP if SCTP has in fact failed, rather than because an attacker
wants it to look like it has failed.  (short of a congestion attack that
would also make UDP fail unless one could control the timing of the end of
the attack down to the millisecond, which seems like a high enough bar.)

so at the moment i think SCTP will be possible.  interestingly, the same
set of middleboxes and firewalls and NATs and PBRs that will prevent SCTP
from working are also preventing EDNS from working right now.  so while we
can legitimately worry that SCTP will be hard to deploy, it turns out that
we have already bitten off a similar challenge.  SCTP has an advantage over
EDNS/UDP in that no middlebox currently knows to look for DNS SCTP messages
and rewrite them.  if we do our job well and quickly, there never will be.

---

From: Douglas Otis <[email-masked]>
Date: Tue, 9 Jun 2009 16:55:20 -0700

On Jun 9, 2009, at 1:23 PM, Matthew Dempsky wrote:

>> To keep it brief, the summary is that SCTP uses at least twice as  
>> many packets as UDP, and far more if the association is not kept  
>> open for a long time. Furthermore, an open association consumes  
>> server resources, and probably not less than TCP (even while  
>> sharing a file descriptor between queries). The initial cookie does  
>> not mitigate this - every association carries state that has to be  
>> carried around.
>
> I don't have any SCTP experience, but these results seem  
> discouraging for DNS-over-SCTP adoption.  I'd be interested in  
> knowing if these results contradict any DNS-over-SCTP proponents'  

Without question, SCTP requires more resources than UDP in the ideal  
case.  However, the ideal case is _not_ where SCTP offers benefit.

Don't compare SCTP in terms of UDP packets used under ideal  
conditions.  Compare SCTP in terms of deterministic defenses against  
both brute and spoofed source attacks, and the number of servers  
required to withstand them.  Defending against even a moderate attack  
requires substantial resources.  In the case of UDP, these resources  
can be directed toward other victims.  In the case of TCP, resources  
might be consumed rapidly by spoofed sources.

With DNS over UDP, there is no way for DNS to prioritize responses or  
to determine with certainty which queries have spoofed source IP  
addresses or which are truly abusive.  In the case of SCTP, a spoofed  
victim should be able to ignore, as a minor nuisance, unrequested SCTP  
cookies.  With SCTP, DNS servers should also be able to dynamically  
exclude excessive queries without unfairly affecting service to those  
not involved.  In the case of DNS over UDP, only BCP38 mitigates most  
traffic to spoofed victims.  Unfortunately, BCP38 is not complete and  
is not a filter instantiated by victims to protect themselves.  Unlike  
SCTP cookies or TCP SYN-ACKs, the overall size of a response to  
spoofed DNS over UDP can create sizable amplifications that grow even  
larger with EDNS/DNSSEC.  In addition, spoofed UDP DNS queries will  
not respond to network congestion.  Brute force reflected attacks  
using EDNS also introduce a potential for induced packet  
fragmentation.  Fragmentation further increases the impact of an  
attack vector.

With DNS over TCP, a transmission control block (TCB) is consumed by  
every SYN received and is retained until it times out.  The rate of  
TCB consumption is not regulated by source IP addresses since these  
address are not confirmed prior to TCBs being committed.  For targeted  
attacks, specific TCB associations having 20 second timeouts might be  
rapidly consumed by a single T1 link.  Whether brute force, reflected,  
or source spoofed, the use of cookies by SCTP provides a method to  
construct fair and automated defenses that are otherwise impractical.

As the speed of the Internet increases, many seem unaware of the  
looming problems ahead.  Many continue to publish wildcard TXT SPF  
records that carry cached macros.  Some now advocate the use of MX 0 .  
records to refute outbound email.  Some have suggested packet  
cryptography offers protection from spoofed or brute force attacks.   
At least SCTP limits attack mitigation to the validation of one's own  
cookie.  Don't just consider the ideal case, consider those cases that  
represent an existence within today's Internet

-Doug

---

From: bert hubert <[email-masked]>
Date: Wed, 10 Jun 2009 20:45:40 +0200

> the association open and use it for multiple transactions.

For it to be useful, we'd need to use it for all our traffic.

> they do appear on the wire.

Between a single auth server and a single cache, <1 QPS will probably
be quite common. This brings into perspective packet counts that are
at least twice as high as UDP, and more likely five times as high
(when taking into account the setup and teardown).

> the DNS server's system call interface (think of a select() mask here, or

Underneath the hood, the kernel is dealing with the exact same
challenge as select() or poll() however. So you just moved the
problem. In addition, it appears that except for only using a single
fd, there is nothing 'lightweight' about an SCTP association.

I measure around 8 kilobytes per association. This would mean your
millions of associations mentioned above require tens of gigabytes of
ram on a commonly used operating system.

SCTP is not remotely like UDP.

> the transport will tell the receiver the length of each incoming message.

The two bytes won't save us however.

> the attack down to the millisecond, which seems like a high enough bar.)

This is exactly how one would use EDNS PING.

So please color me unconvinced. SCTP has its merits, but it is not
light weight, and uses a lot of packets compared to UDP or even TCP.

         Bert

---

From: bert hubert <[email-masked]>
Date: Wed, 10 Jun 2009 20:42:46 +0200

Bert

---

From: niels=[email-masked] (Niels Bakker)
Date: Wed, 10 Jun 2009 23:59:45 +0200

> case.  However, the ideal case is _not_ where SCTP offers benefit.

Can you describe your idea of the average case, then?  It would be a 
huge improvement over your current hand-waving

In case you disagree with the data bert hubert presented, the answer 
should be more data

>represent an existence within today's Internet

What does "IN MX 0 ." have to do with what transport is used for sending 
DNS datagrams across?

	-- Niels.

--

---

From: Douglas Otis <[email-masked]>
Date: Wed, 10 Jun 2009 17:25:05 -0700

On Jun 10, 2009, at 2:59 PM, Niels Bakker wrote:

>> Without question, SCTP requires more resources than UDP in the  
>> ideal case.  However, the ideal case is _not_ where SCTP offers  
>> benefit.
>

DNS represents an essential service that needs to remain operational  
during at least moderate levels of attack.  Daily bot-net activity  
revealed over a few trillion DNS transactions in our tiny network show  
different quantitative results over increased timeframes.  The overall  
sources have been steadily doubling, as if governed by Moore's law.   
Nefarious activity seen through the straw-like-view from our tiny  
network appear to emanate from a few million IP addresses within an  
hour, and grow to hundreds of millions over a few weeks with an  
asymptote at about two years.  While seldom has the activity been  
aimed at causing DDoS events, in our defense against occasional  
attacks that appear aimed at enhancing malware spread, large  
expenditures become required.  Unfortunately, management has decided  
not to make details public.  Much of the attack appears to represent  
re-tasking of the bot-net drones.

> In case you disagree with the data bert hubert presented, the answer  

Unfortunately, much of this data will need to be extrapolations of  
attacks permitted by today's networks.

>> records that carry cached macros.  Some now advocate the use of MX  
>> 0 .  records to refute outbound email.  Some have suggested packet   
>> cryptography offers protection from spoofed or brute force  
>> attacks.   At least SCTP limits attack mitigation to the validation  
>> of one's own  cookie.  Don't just consider the ideal case, consider  
>> those cases that  represent an existence within today's Internet
>
> What does "IN MX 0 ." have to do with what transport is used for  
> sending DNS datagrams across?

These records are likely to generate additional junk traffic from  
millions of otherwise innocent sources that will be targeting roots  
with fairly short negative caching.  The traffic created by this, in  
conjunction with DNSSEC, IPv6 and EDNS will raise the base level that  
roots will need to support.

-Doug

---

From: Shane Kerr <[email-masked]>
Date: Thu, 11 Jun 2009 11:47:15 +0200

Bert,

> ram on a commonly used operating system.

AIUI, SCTP can support reliable, ordered delivery like TCP, or it can
run without those features. The protocol stack will require memory to
track the state of things if those are enabled, which it will not
require if these are not used.

So, hopefully those 8 kilobytes per association can be greatly reduced
if we only need the UDP guarantees (meaning - "nothing"). :)

> light weight, and uses a lot of packets compared to UDP or even TCP.

There is one more packet required for the initial handshake than for
TCP. Since data can be present in the same packets as control
information, the actual sequence for a complete SCTP session might be:

Client -> INIT
          INIT-ACK     <- Server
Client -> COOKIE-ECHO                # put query here   
          COOKIE-ACK   <- Server     # put response here
Client -> SHUTDOWN                   
          SHUTDOWN-ACK <- Server
Client -> SHUTDOWN-COMPLETION

So we've turned a 2-packet transaction UDP transaction, or a 6-packet
TCP transaction, into a 7-packet SCTP transaction.

The latency from the client point of view is 2 RTT (the time until the
COOKIE-ACK comes back with the query response) - not great but not too
horrible.

I don't know what the relative bandwidth consumption is going to be of
the two streams. It will be more, obviously, but not 3.5x since the
packets with only control data should be quite small.

Also note that the most common DNS query is likely to be A followed by
AAAA, or possibly the other way around(*). A smart client will keep the
session open, meaning the total packet count would be 4 for UDP, 8 for
TCP, and 9 for SCTP. The total latency will be 2 RTT for UDP and 3 RTT
for TCP or SCTP.

Going further to the case where we are looking at the communication
between stub resolvers and recursive resolvers, then if stub resolvers
are able to keep the session open without huge cost on the recursive
resolver, then the amortized cost will be the same for SCTP as for UDP -
2 packets per lookup.

--
Shane

(*) Maybe it makes sense to create an EDNS0 option to include 
    additional RTYPEs in a single query?

---

From: Douglas Otis <[email-masked]>
Date: Thu, 11 Jun 2009 16:56:28 -0700

On Jun 11, 2009, at 1:13 AM, Florian Weimer wrote:

> * Paul Vixie:
>
>> it takes two packets to carry a request and a response, and if they  
>> are the last request and response in the queue then each of these  
>> will be followed by an ACK packet.  so, a slow 1TPS flow would take  
>> four packets per transaction, whereas a faster flow would take two  
>> packets per transaction.  the extra ACK-only packets don't delay  
>> the DNS result, but they do appear on the wire.
>
> If you use SCTP's retransmit feature, you need in-kernel data  
> buffers or delays in user space.  A new PR-SCTP service might work  
> around this problem, but it seems to me that some protocol work is  
> required.

PR-SCTP will mean retries are handled by a layer above the transport,  
just as they are for UDP.  For servers, this might conserve  
resources.  Using SCTP_UNORDERED should minimize head of queue  
blocking and free buffer space sooner.  It would be nice to have  
various access statistics available to assess various strategies that  
might be used when employing persistent associations.  IIRC, the  
default number of persistent associations supported by the FreeBSD  
kernel is 40K.  Many of our customers access our DNS through an ISP's  
resolver, where there is more than a two to one reduction of sources  
for records having 300s TTLs.  For our application, no modification of  
FreeBSD would appear needed.  The duration of an inactive association  
could be regulated by the number of available associations.

>> by lightweight i mean that a root or TLD or 2LD/3LD server could  
>> have many hundred thousands or even millions of associations open,  
>> all on a single socket and therefore a single file descriptor, so  
>> it's not a burden on the DNS server's system call interface (think  
>> of a select() mask here, or a poll() filedes array.)
>
> If you keep that many idle associations open, your servers will be  
> busy processing all the associated heartbeat messages.

To prevent resource exhaustion whenever multihoming is used, an  
initial heartbeat confirms alternate paths.  For on-going  
associations, heartbeat rates can be adjusted or turned off.

> The one-to-many style interface also needs in-kernel buffers, and it  
> will be somewhat difficult to control the amount of resources used  
> by busy resolver.  Furthermore, it is explicitly left open in the  
> sockets API draft whether one receiver behind a one-to-many socket  
> can block communications to other receivers (see section 4.4 in  
> draft-ietf-tsvwg-sctpsocket-19).

The API supports moving a one-to-many association to a different one- 
to-one descriptor and establish independent outbound allocations.   
There is also an SCTP_AUTOCLOSE that might take care of most  
association housekeeping.  The goal should be to retain the most  
active associations as persistent to ensure minimal latency and  
somewhat reduced traffic.  SCTP's source validation eliminates  
concerns related to reflected attack, where any increase in traffic  
should appear small when compared to everything else now going over  
the Internet.

-Doug

---

From: Danny Mayer <[email-masked]>
Date: Sat, 13 Jun 2009 22:24:13 -0400

> some 3LD's also need, this whole thread is "twilight zone" territory.

But isn't a signed root zone going to cause problems for such providers?
They are presumably changing the contents and therefore the signature
will no longer match the contents and you will likely get dnssec
failures. After all isn't that the intent of DNSSEC signing of the root
zone? If they go ahead with their own resigning will that "fix" the
problem or continue to invalidate all the signed zones down the hierarchy?

Danny

---

From: Doug Otis <[email-masked]>
Date: Wed, 10 Jun 2009 14:39:09 -0700

> For it to be useful, we'd need to use it for all our traffic.

Persistence of an association can be dictated by availability of  
resources and frequency of use.

> Between a single auth server and a single cache, <1 QPS will  
> probably be quite common. This brings into perspective packet counts  
> that are at least twice as high as UDP, and more likely five times  
> as high (when taking into account the setup and teardown).

UDP does not permit any easy mitigation of brute force or spoofed  
source attacks where packets are larger or fragmented due to EDNS.   
Any practical defense leaves TCP or SCTP.

> fd, there is nothing 'lightweight' about an SCTP association.

Unlike TCP, resources are not committed prior to confirmation of the  
source IP addresses needed to mitigate flood attacks.  There is  
nothing lightweight about the alternative.  In addition, buffer  
management is made simpler by specific acknowledgements and out of  
order delivery, rather than buffering to a fully acknowledge byte  
position within a common stream.  Buffer management is also simplified  
by transport header alignment with DNS messages that enables UDP like  
immediate placement upon receipt, despite pending prior packet loss.

> millions of associations mentioned above require tens of gigabytes  
> of ram on a commonly used operating system.

This resource can allocated between trusted and high traffic servers.   
Low traffic sources might be required to establish associations.

> SCTP is not remotely like UDP.

Independent acknowledgement and out of order handling of SCTP chucks  
(DNS messages) is much closer to UDP behaviors than TCP would be.

> The two bytes won't save us however.

AUTH-SCTP can introduce a 32 byte nounce.  However, DNSSEC does not  
depend upon transactional ids once all cache information is being  
confirmed.

> light weight, and uses a lot of packets compared to UDP or even TCP.

SCTP offers defensive strategies not practical with TCP.  UDP creates  
different risks that grow with EDNS/DNSSEC.

The comparative overhead should include additional servers needed to  
endure attack.  Once that consideration is included, a moderate  
increase in packet volume during ideal conditions becomes much less  
pronounced as a concern.

-Doug

---

From: Florian Weimer <[email-masked]>
Date: Thu, 11 Jun 2009 10:13:00 +0200

> they do appear on the wire.

If you use SCTP's retransmit feature, you need in-kernel data buffers
or delays in user space.  A new PR-SCTP service might work around this
problem, but it seems to me that some protocol work is required.

The one-to-many style interface also needs in-kernel buffers, and it
will be somewhat difficult to control the amount of resources used by
busy resolver.  Furthermore, it is explicitly left open in the sockets
API draft whether one receiver behind a one-to-many socket can block
communications to other receivers (see section 4.4 in
draft-ietf-tsvwg-sctpsocket-19).