Subject: Re: [dnsext] draft-ietf-dnsext-dns-tcp-requirements-00 / a way forward

From: Alfred Hönes <[email-masked]>
Date: Wed, 14 Oct 2009 20:41:45 +0200 (MESZ)

> Dear all,
>
> sorry for answering late, but I was traveling...
> Some comments in-line.
>
> After looking at
> draft-ietf-dnsext-dns-tcp-requirements-00
> I think you want to use an unmodified TCP. So deployment
> of the transport stack is not an issue. Am I right?
>
> More comments in-line.
>
> Best regards
> Michael
>
> ...[snip]...

Michael, Andrew, all,

Focusing on draft-ietf-dnsext-dns-tcp-requirements-00,
I would like to reinforce that I had never proposed to include
discussion of 'other DNS transports' into that document.

My suggestion was to elaborate on TCP properties for several
reasons (see below), and for the WG to obtain support from
transport protocol experts in pursuing its chartered work item
to evaluate existing and consider new transports for DNS.

I admit that the nearby publication of the above draft and
George Barwoods DNS transport related drafts, as well as the
rechartering of the WG may have lead to comments with a
Subject not totally encompassing some ideas contained in the
message that had been cross-fertilized by these (natively
independent) events.

So let's focus on TCP.

I fully support the idea of the draft, as already indicated.

And I fully support Michael in that we want to use "unmodified TCP".

But TCP has a lot of Options, and perhaps all commercial and
open-source TCP stack implementations have a lot of tuning knobs
to adjust parameters, enable/disable options and mechanisms, etc.
-- that's particularly interesting for DNS server/resolver operators.
Furthermore, the TCP (socket) API contains features that allow
programmatic access to particular features, mechanisms and options
of TCP -- how to best make use thereof is highly interesting for
implementers of DNS software.

The main reason for having this draft is the perceived unwillingness
of various parties to implement/support/allow TCP as it was intended,
a mandatory transport for DNS in case UDP does not suffice, in
particular for packet size / fragmentation reasons.

We (or many of us) want to convince those parties posing obstacles
to this use of TCP that their opposition is not justified, and that
IMO cannot be done by a simple normative statement of "MUST" for TCP.
No, we need to provide arguments and show ways to mitigate the raised
concerns!  Exploring the effects of TCP options and parameters and
giving recommendations to implementers and operators of how to use TCP
in the best possible manner seems the right way to achieve this goal.
This should include concrete arguments for implementors and vendors
of middleboxes that DNS over TCP is not evil.

If we can agree on this strategy, the draft has a chance to be
amended and turned into a valuable document that gives enough
concrete guidance to both implementers and operators to actually
overcome the observed reservedness against the TCP support for
DNS transactions as specified in STD 13 and STD 3.

Such exploration and elaboration should also better educate the
WG on aspects of DNS transport, allowing it to perform -- in a
second step then -- a more targeted evaluation of proposals for
alternate DNS transport(s), and eventually make better informed
decisions on how to proceed with new proposals to this end.

N.B.: The priority of dealing with possible signalling of Transport
(and/or other feature) support in DNS servers may very well be made
dependent on the outcome of the first steps above.

Can wee agree on this way forward?

Kind regards,
  Alfred Hönes.

-- 

+------------------------+--------------------------------------------+
| TR-Sys Alfred Hoenes   |  Alfred Hoenes   Dipl.-Math., Dipl.-Phys.  |
| Gerlinger Strasse 12   |  Phone: (+49)7156/9635-0, Fax: -18         |
| D-71254  Ditzingen     |  E-Mail:  [email-masked]                     |
+------------------------+--------------------------------------------+

---

From: [email-masked]
Date: Wed, 14 Oct 2009 20:16:06 +0100

> And I fully support Michael in that we want to use "unmodified TCP".

Indeed - modifying TCP itself is not (as far as I'm concerned) on the 
table.

> implementers of DNS software.

It certainly is.

FWIW, yesterday I performed some benchmark tests against Bind 9.6.

Over TCP/IP I was able to do 3000 qps.  That was with a brand new TCP 
connection for each query, with no parallelisation.  The client code was 
literally:

while (1) {
  socket(...);
  connect(...);
  write(2 byte len);
  write(request)
  read(2 byte len);
  read(response);
  close();
}

The same server will do about 23000 qps over UDP.

Personally I think that TCP performance is pretty good.  I expect that 
with more queries in flight at the same time the effective performance 
might be double that at least.

[BTW, the TCP stack on the _server_ didn't suffer at all, as it was the 
client end that was actively closing the connection.  However the client 
very rapidly ran out of ephemeral ports because of the quantity that were 
in TIME_WAIT state].

> particular for packet size / fragmentation reasons.

Exactly, yes.

> of middleboxes that DNS over TCP is not evil.

This is what I've attempted to do already, but I accept that it needs more 
work.

Mostly what I'd like to be able to do is dispel (or at least mitigate) the 
FUD about TCP as a transport mechanism for DNS.

> Can wee agree on this way forward?

This sounds sensible, although I would like to take the counsel of those 
wiser to the ways of the IETF than I am.

At the very least, the idea of having a real expert on TCP advise is very 
sensible.

kind regards,

Ray

-- 
Ray Bellis, MA(Oxon) MIET
Senior Researcher in Advanced Projects, Nominet
e: [email-masked], t: +44 1865 332211

---

From: Paul Vixie <[email-masked]>
Date: Wed, 14 Oct 2009 20:12:16 +0000

> From: [email-masked]
> Date: Wed, 14 Oct 2009 20:16:06 +0100
> 
> ... - modifying TCP itself is not (as far as I'm concerned) on the table.

for the purpose of this draft i think that's a safe assumption.

> might be double that at least.

what would the performance be if the server stopped receiving client FINs?

> Mostly what I'd like to be able to do is dispel (or at least mitigate)
> the FUD about TCP as a transport mechanism for DNS.

"bring it on."  :-)

---

From: Douglas Otis <[email-masked]>
Date: Wed, 14 Oct 2009 21:54:48 -0700

On 10/14/09 12:16 PM, [email-masked] wrote:
> connection for each query, with no parallelisation.

This has ignored loads that might be created by bad actors with the goal 
of burdening servers.  Transactions per second statements should also 
cover the level of reduction caused by credible attack.

> The same server will do about 23000 qps over UDP.

This would seem to imply that after an 8x increase in the number of 
servers, DNS over TCP could then achieve its current margins.  Even 
after such a sizable investment, retaining current margins seems 
implausible.

>  > particular for packet size / fragmentation reasons.

DNS is not currently dependent upon TCP, nor had DNS been designed to 
become dependent upon TCP.  The fact that most CPE equipment does not 
proxy DNS over TCP should give this WG some pause about the economical 
practicality of depending upon TCP.

At least, TCP will ensure DNS represents less of a threat when exploited 
to attack other services.  However, nearly all other services heavily 
depend upon DNS remaining operational.

> This is what I've attempted to do already, but I accept that it needs
> more work.
>

Would you suggest a substantial increases in the resources needed to 
sustain current levels of service is not cause for concern or careful 
review of possible alternative?  It seems important to ensure that 
resources estimates be based upon reasonable expectations of what might 
be needed to endure credible levels of abuse?

> At the very least, the idea of having a real expert on TCP advise is
> very sensible.

One might have hoped TCP would _not_ have been seen as the _only_ option 
available, and that fair comparisons of the alternatives would have been 
sought instead of discouraged.  It seems unwise to eliminate 
alternatives already providing service for other critical infrastructure 
in a manner similar to that required by DNS.

When dependent upon TCP, how much DNS infrastructure will become 
dependent upon Akamai like services, and who might desire this outcome?

-Doug

---

From: [email-masked]
Date: Thu, 15 Oct 2009 07:24:55 +0100

> cover the level of reduction caused by credible attack.

Doug,

Indeed, that was a lab test.  As Vix indrectly pointed out, a client that 
fails to send FIN packets is not good for servers.

However, 90% of the TLD authoritative servers and all but 'm' from the 
root servers *already* support TCP/IP.

And yet the DNS is not collapsing under the weight of DoS attacks against 
them.

> practicality of depending upon TCP.

The fact that very few middlebox vendors implement TCP is not, IMHO, 
because of some deeply held reservations about TCP.  Hell, many such boxes 
proxy HTTP without any problem and that's far harder.  As far as I can 
tell it's simple laziness on the part of the implementors.

Ray

---

From: Doug Otis <[email-masked]>
Date: Thu, 15 Oct 2009 11:11:44 -0700

On 10/14/09 11:24 PM, [email-masked] wrote:
>> This has ignored loads that might be created by bad actors with
>> the goal of burdening servers. Transactions per second statements
>> should also cover the level of reduction caused by credible
>> attack.
>
> Doug,
>
> Indeed, that was a lab test. As Vix indrectly pointed out, a client
> that fails to send FIN packets is not good for servers.
>
> However, 90% of the TLD authoritative servers and all but 'm' from
> the root servers *already* support TCP/IP.
>
> And yet the DNS is not collapsing under the weight of DoS attacks
> against them.

Be careful with conclusions based upon TCP availability versus TCP
dependence. Once DNS becomes dependent upon TCP operations as a result
of increased message size and the deprecation of EDNS0 to handle these
messages, then resource sharing with TCP may need adjusting.  A
preference for UDP and lack of dependence upon TCP could be why DNS
remains functional.

>> DNS is not currently dependent upon TCP, nor had DNS been designed
>> to become dependent upon TCP. The fact that most CPE equipment
>> does not proxy DNS over TCP should give this WG some pause about
>> the economical practicality of depending upon TCP.
>
> because of some deeply held reservations about TCP. Hell, many such
> boxes proxy HTTP without any problem and that's far harder. As far as
> I can tell it's simple laziness on the part of the implementors.

For HTTP proxies, a critical operational requirement is sufficient
memory resources.  Reductions in network demand or security concerns
help justify these resources. Network reductions will not occur for DNS,
nor will DNS proxies improve security.  The same economics that
determine whether TCP is used in CPE equipment for DNS likely represents
a microcosm of DNS resource economics.  Needing more resources to offer
the same service seems destine to produce a trade-off in reduced
reliability.

-Doug

---

From: Alex Bligh <[email-masked]>
Date: Thu, 15 Oct 2009 19:45:57 +0100

> preference for UDP and lack of dependence upon TCP could be why DNS

Mmmm... there seems to be a horse/cart issue here. DNS is not functional in
practice, in that it does not reliable transport query responses that are
needed to transmit information securely. I will avoid putting words in your
mouth Doug, as I don't think you've gone this far, but if someone were to
argue, "don't make tcp support server side mandatory because if everyone
used it the servers might melt" (and I note the likelihood of this is not
yet established), my response would be that this is like suggesting we
avoid querying web-servers in case we overload them. Just like http, the
DNS protocol is being asked to do more and more work per query as query
sizes increase, and more queries as total users increase. Keeping the
protocol crippled so it can't transmit large packets in order to avoid
servers being overloaded would be a peculiar option.

There may well be tweaks to TCP, tweaks to TCP parameters, or entirely new
transport protocols which would be far better. However, as far as I know,
they are not yet ready to deploy. Thus they aren't reasons not to make TCP
support mandatory.

What, beyond a warning that TCP support may lead to substantial resource
requirements on servers, are you suggesting in terms of this draft
(if anything)?

-- 
Alex Bligh

---

From: [email-masked]
Date: Thu, 15 Oct 2009 19:50:33 +0100

> preference for UDP and lack of dependence upon TCP could be why DNS

I haven't actually heard anyone suggesting that we deprecate EDNS0.  I 
don't think my draft can be interpreted as suggesting it either.  *Most* 
of the time it works.  In any event EDNS0 is currently required even in 
TCP to handle the DO bit.

The last figures I heard from signed zones were that TCP volumes were ~1% 
of UDP volumes.  My initial testing does suggest that that level is easily 
manageable, although as yet I haven't done enough testing to actually 
prove it.

Ray

---

From: Paul Vixie <[email-masked]>
Date: Thu, 15 Oct 2009 21:11:50 +0000

> Date: Thu, 15 Oct 2009 07:24:55 +0100
> ...

it would become so, the moment we started to depend on tcp as a primary
delivery channel rather than as an occasional fallback.  if you give an
attacker a knob they might not twist it.  if twisting the knob would make
your primary service unavailable to others, then they absolutely will twist
it.  so i'll say again: if we *require* state, then the whole system will
become more vulnerable and less robust, unless that state is defended much
better than can be done with TCP as it is presently defined.

---

From: Doug Otis <[email-masked]>
Date: Thu, 15 Oct 2009 15:49:45 -0700

> TCP to handle the DO bit.

Sorry, the deprecation of large frames to handle UDP for DNSSEC.  This 
might be the result when EDNS0/UDP exploitation grows worse as DNSSEC 
becomes prevalent.

> The last figures I heard from signed zones were that TCP volumes were
> ~1% of UDP volumes. My initial testing does suggest that that level is
> easily manageable, although as yet I haven't done enough testing to
> actually prove it.

The impetus for a TCP mandate is to offer an assured alternative to UDP 
when handling larger frames. The percentage of systems dependent upon 
TCP might then change dramatically, especially if UDP proves problematic 
with these larger frames.  Mandating TCP support implies TCP will remain 
dependable. If or when TCP becomes prominently used for general queries 
in support of DNSSEC, DNS reliability will have been significantly 
eroded.  Only transports able to sustain the DNS service when faced with 
abuse should be mandated and thereby assured.

It will not benefit TCP to have its API dramatically changed, or to have 
it defined as an object rather than a byte stream.  There are other 
transports in use for nearly a decade able to fulfill this role by 
offering unreliable unordered delivery of objects.  This allows server 
state to exclude in-flight data and to be limited to socket tuples, 
tags, and object counts following a cookie exchange.  The cookie 
exchange is needed as insurance against reflected attacks without 
reliance upon BCP38.

-Doug

---

From: Mark Andrews <[email-masked]>
Date: Fri, 16 Oct 2009 10:13:12 +1100

> better than can be done with TCP as it is presently defined.

What people are demanding is that the occasional fallback channel
actually be available when it is needed.  That is a reasonable
requirement.

There are too many lazy vendors in the DNS marketplace, we wouldn't
have the problems we do with load balancers and proxies if the
vendors were actually conscientious and looked at the RFCs.

As far as I can tell both classes of vendor just looked at the
packets usually on the wire and said that's all we need to support
and forgot about the rest.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Alex Bligh <[email-masked]>
Date: Fri, 16 Oct 2009 00:13:18 +0100

> might be the result when EDNS0/UDP exploitation grows worse as DNSSEC

May be the draft should make clear that UDP is still mandated as well.

I can't really see why TCP would be used in preference to UDP when there
is no middlebox in the way. Even if all middleboxen were evil, and can't
be worked around (e.g. through proxy-bypass), recursive servers would
still use UDP to make their recursive queries, so I would not expect
TCP load on authoritative servers to increase nearly as much.

Clients of recursive servers can presumably take advantage of
persistent connections (and recursive servers can favour clients
that do so).

-- 
Alex Bligh

---

From: Doug Otis <[email-masked]>
Date: Thu, 15 Oct 2009 20:20:32 -0700

> May be the draft should make clear that UDP is still mandated as well.

A UDP mandate to handle what exactly?  Should EDNS0/UDP be required to 
handle 1280, 1480, 4096, 8192, 9216 or perhaps 64k bytes?  What minimum 
frame size is required to dissuade TCP use in an era of DNSSEC?  What 
happens when reflected attacks cause large DNS UDP frame support to be 
dropped, especially after TCP support has been mandated?

Mandating TCP support for DNS will be misleading, as this suggests TCP 
represents a suitable replacement for UDP.  It does not.  It would also 
be wrong to insist TCP represents a suitable replacement for UDP simply 
because it will only be used in a small percentage of the transactions. 
  Any such assumption is likely to be proved wildly inaccurate in short 
order.  Especially when the costs of TCP resources are much less for 
consumers than for those attempting to retain a robust service.

> TCP load on authoritative servers to increase nearly as much.

There are DNS reflected attacks being staged that take out multiple 
OC192 connections.  This type of threat with the introduction of DNSSEC 
may play a role in the deprecation of large frame UDP DNS.  After all, a 
mandate for TCP implies there is no reason to be concerned.  Just use 
TCP right?

> persistent connections (and recursive servers can favour clients

TCP servers will still be required to retain unacknowledged data, where 
persistence does not seem to help in reducing this exposure.  TCP takes 
DNS in the wrong direction.

-Doug

---

From: Mark Andrews <[email-masked]>
Date: Fri, 16 Oct 2009 15:03:00 +1100

In message <[email-masked]>, Doug Otis writes:
> represents a suitable replacement for UDP.

No. It doesn't.  It reflects the reality TCP has always been required
by general purpose resolvers not operating in a highly controlled
environment that prohibited UDP responses that exceed 512 bytes.

Mark

---

From: Doug Otis <[email-masked]>
Date: Thu, 15 Oct 2009 21:45:29 -0700

On 10/15/09 9:03 PM, Mark Andrews wrote:
> In message<[email-masked]>, Doug Otis writes:
>

Mark,

TCP is _not_ always required.

-Doug

---

From: Mark Andrews <[email-masked]>
Date: Fri, 16 Oct 2009 16:20:28 +1100

> TCP is _not_ always required.

Please re-read what I said.

---

From: [email-masked]
Date: Fri, 16 Oct 2009 08:32:36 +0100

> actually be available when it is needed.  That is a reasonable

Absolutely.  For _most_ large authoritative and presumably recursive 
installations TCP is already there.  This draft is trying to catch the 
edge cases, and is very unlikely _on it's own_ to make any significant 
difference to TCP query volumes.
 
> and forgot about the rest.

That's exactly my view as well.

Ray

---

From: Paul Vixie <[email-masked]>
Date: Fri, 16 Oct 2009 07:47:44 +0000

> Date: Fri, 16 Oct 2009 08:32:36 +0100
> 

does the current draft make it clear that tcp is (and has always been)
mandatory AND that it is (and has always been) a backup or secondary
transport (after UDP/EDNS)?

if so then i withdraw any objection i was thinking i might have to it.

> > have the problems we do with load balancers and proxies if the vendors
> > were actually conscientious and looked at the RFCs.
> > 
> > As far as I can tell both classes of vendor just looked at the packets
> > usually on the wire and said that's all we need to support and forgot
> > about the rest.
> 

and mine.

---

From: [email-masked]
Date: Fri, 16 Oct 2009 08:59:12 +0100

> delivery channel rather than as an occasional fallback.

I'm not proposing that it become the primary channel, although if it helps 
I might considering watering down the "you don't always have to do UDP 
first" section.

> if you give an

I'll say again: that knob is _already_ available to twist on the vast 
majority of DNS installations.  The vendors of their software (mostly ISC 
BIND, of course) and their operators already know how to read RFCs and 
support TCP by default.

It's the middleboxes sat in the path and other minority edge cases that 
I'm interested in.

Ray

---

From: [email-masked]
Date: Fri, 16 Oct 2009 09:05:49 +0100

> transport (after UDP/EDNS)?

It's not quite as explicit as that.

It points out that in 1035 and 1123 it was "SHOULD", but that 1123 
predicted a time when it would become "required" (non-2119 required).

So this draft changes that into "REQUIRED" except in a small set of edge 
cases.

It does not attempt to make TCP a primary transport, with the caveat that 
it says: "A resolver SHOULD send a UDP query first, but MAY elect to send 
a TCP query instead if it has good reason to expect the response would be 
truncated if it were sent over UDP, or for other operational reasons."

Ray

---

From: Alex Bligh <[email-masked]>
Date: Fri, 16 Oct 2009 09:56:37 +0100

> So this draft changes that into "REQUIRED" except in a small set of edge

I think this is a wise course to steer, not least because there was
a debate on here as to whether MUST'ing TCP support was a clarification
(as per the original work item), or a marginal change in requirements,
and that debate didn't appear to reach consensus. Given that almost
all those who thought it was a marginal change believed that support
(under most circumstances) should be a MUST anyway, discussing in
exactly what circumstances TCP support is currently a SHOULD seems
a little bit of an angels-on-pinheads argument, as the very presence of
the work item illustrates that not everyone agrees what the current
requirement level for TCP support is.

-- 
Alex Bligh