Subject: [dnsext] all .gov do=1 nxdomain answers already appear to be fragmented.. and it works somehow

From: bert hubert <[email-masked]>
Date: Tue, 18 Aug 2009 23:54:41 +0200

On Tue, Aug 18, 2009 at 11:12 PM, Andrew Sullivan<[email-masked]> wrote:
> No hat.
> I think that's true.  The important question to ask, for deployment
> purposes (and I suggest the answer to this belongs on another list),
> is what "it breaks" means.

Perhaps - but from the 'theoretical side', we should be aware of what
we have wrought.
Oddly enough, some very common (.gov) responses are fragmented right
now, and we don't see issues.

The two biggest typical DNSSEC responses appear to be:

1)  the one with the DNSKEY (that could, btw, travel safely over TCP,
since it should be none too frequent, and the validating resolver
*knows* it might be a whopper). This includes 'ANY' queries, which
could well return DNSKEY. .GOV has this one at 1166 bytes, .ORG  at
1723 and .SE at 4 kilobytes.

2) an NSEC3 NXDOMAIN (which comes in unexpectedly)

Response 1 is the least scary, since it should only be performed by
people that have actively turned on DNSSEC validation, or who are
doing relatively rare ANY queries. If you can't deal with the (huge)
size of this response, your normal DNS will work, but nothing will
validate.

Response '2' hits all recent BINDs that in their pioneering wisdom
decided to set do=1 on outgoing queries. If that one gets too big, all
BIND users will instantly have a problem. I actually & honestly do
think ISC did us a service setting do=1 by default btw. Better suffer
now than discover a problem later.

Impact
----------
Response 1 is the biggest, but only impacts 'DNSSEC pioneers', and
they could move to TCP for this query.

Response 2 appears to be relatively harmless, and I don't understand
why. Almost all .GOV do=1 NXDOMAIN answers are fragmented right now.
And I don't hear people complaining, so there may be some hope yet.

Nicholas, any ideas? Or is BIND being really smart with fallback? Are
any of the .GOV people at liberty to share their experiences?

Response '2' might still very well kill DNSSEC for a significant
percentage of people *or zones*, which would lead to the
aforementioned 'I turned it on and stuff broke' behaviour. This pain
will be borne whenever someone DNSSEC signs his zone, and someone
behind a validating resolver (with a trust anchor) tries to access
that zone.

23:43:30.421519 IP 10.0.1.47.58971 > 76.74.254.208.53: 56855+ [1au] A?
www.c1a.gov. (40)
23:43:30.590564 IP 76.74.254.208.53 > 10.0.1.47.58971: 56855
NXDomain*- 0/8/1 (1472)
23:43:30.590624 IP 76.74.254.208 > 10.0.1.47: udp
;; MSG SIZE  rcvd: 1503

    Bert

---

From: Paul Wouters <[email-masked]>
Date: Tue, 18 Aug 2009 18:31:38 -0400 (EDT)

> And I don't hear people complaining, so there may be some hope yet.

What kind of complaint were you expecting from people trying to
reach doesnotexist.gov ? :)

> behind a validating resolver (with a trust anchor) tries to access

I've been running DNSSEC on my zones for years. I've never hear someone
complain they couldn't reach my sites. I've been running dnssec on my
resolvers with all known trust anchors loaded and DLV enabled for about
a year now, and have only seen a few problems, and those mostly related
to enabling 0x20 and wijngaards-dnsext-resolver-side-mitigation, not
dnssec in itself.

If you have a clean network, these things do "just work".

And it is not about when "we" enable dnssec. dnssec is already enabled by
enough people to know that by Toutatis, the sky is not going to fall down.
(for an overview of deployments: http://www.xelerance.com/dnssec/)

Paul

---

From: bert hubert <[email-masked]>
Date: Wed, 19 Aug 2009 00:51:43 +0200

> reach doesnotexist.gov ? :)

Good point.

> If you have a clean network, these things do "just work".

This, however, is not the point; we are not dealing with a 'clean
network'. We are dealing with a network ('The Internet') that saw a
600-fold increase in TCP traffic, which should not have happened.

I understand your stance that we should no longer 'wait' for the
people behind, say, Cisco firewalls. But I'm afraid in reality, it is
not about us waiting for them.

But this might be getting too operational for DNSEXT.

    Bert

---

From: Mark Andrews <[email-masked]>
Date: Wed, 19 Aug 2009 12:02:25 +1000

In message <[email-masked]>, bert 
hubert writes:
> > I think that's true. =A0The important question to ask, for deployment
> decided to set do=1 on outgoing queries.

Well the bit does say "DNSSEC records are OK" and that is true for
all versions of named.

> If that one gets too big, all

Named just falls back to something that works.  It tries EDNS@4096.
If that fails (times out), it tries EDNS@512.  If that fails (times
out) it does plain DNS.  If any of the responses sets TC then named
will retry the query using TCP, using the EDNS state learnt by the
UDP queries, before returning the response to the client.

Note we were overly aggressive in the EDNS fallback and if you want
DNSSEC validation to work reliably in the presence DNS/UDP breaking
middleware you need this bug fix.

2564.   [bug]           Only take EDNS fallback steps when processing timeouts.
                        [RT #19405]
 
If you are not validating then you don't notice this problem.  People
who validate tend to also keep their nameservers current so this
is not a real problem.  By the time DNSSEC becomes general most
vendors should have picked up fixed version of BIND.

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Doug Barton <[email-masked]>
Date: Tue, 18 Aug 2009 22:17:34 -0700

> 600-fold increase in TCP traffic, which should not have happened.

I asked in passing for references to this claim, now I'm asking directly.

Doug

---

From: Mark Andrews <[email-masked]>
Date: Wed, 19 Aug 2009 15:46:35 +1000

> I asked in passing for references to this claim, now I'm asking directly.

The traffic went from .1 connects/second to 60 connections/second compared
to 6000 UDP requests/second.

Mark

---

From: Mark Andrews <[email-masked]>
Date: Thu, 20 Aug 2009 09:42:19 +1000

In message <[email-masked]>, Florian Weimer writes:
> * Mark Andrews:
> For .gov?

ORG.  I don't believe GOV has published statistics.

Mark

---

From: bert hubert <[email-masked]>
Date: Thu, 20 Aug 2009 08:04:52 +0200

On Thu, Aug 20, 2009 at 3:02 AM, Florian Weimer<[email-masked]> wrote:

> Okay, I just had to ask because of the subject line. 8-)

GOV operational numbers would be very interesting given their >1470
NXDOMAIN responses. It would be *really* good if they could share
their experiences (perhaps on dns-operations).

So if anyone has any influence there..

    Bert

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 20 Aug 2009 10:02:04 -0400

No hat.

On Wed, Aug 19, 2009 at 12:02:25PM +1000, Mark Andrews wrote:

> If that fails (times out), it tries EDNS@512.  

I just want to highlight that step 2 is controversial.  In particular,
the claim that EDNS0 at 512 "works" is the one about which people
disagree.  As usual, this all hinges on the value of "works".  The
Internet has made even more true than ever the observation starting,
"The nice thing about standards…."

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 20 Aug 2009 10:31:25 -0400

> their experiences (perhaps on dns-operations).

I have reason to believe (but not personal evidence) that Requests
Have Been Made.  I have no evidence one way or the other whether those
requests will be satisfied.  As list moderator, I agree that an
operational forum may well be the best place to deliver those
statistics.  If the zone operator wanted some control over who got
them, I have a hunch (but I haven't asked) that OARC might be
co-operative.  It would be quite a bit more helpful to the DNSSEC
community, however, if they were published in a more-accessible forum.
Some of us aren't OARC members, and likely don't qualify to be.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Florian Weimer <[email-masked]>
Date: Thu, 20 Aug 2009 03:02:44 +0200

[ Moderators note: Post was moderated, either because it was posted by
   a non-subscriber, or because it was over 20K.  
   With the massive amount of spam, it is easy to miss and therefore 
   delete relevant posts by non-subscribers. 
   Please fix your subscription addresses. ]

---

From: Florian Weimer <[email-masked]>
Date: Wed, 19 Aug 2009 19:08:10 +0200

For .gov?

---

From: Matt Larson <[email-masked]>
Date: Thu, 20 Aug 2009 17:00:57 -0400

On Thu, 20 Aug 2009, Andrew Sullivan wrote:
> co-operative.

Wearing my OARC board member hat, what you are describing is exactly
what OARC was designed to do.  OARC would be happy to accept this data
and redistribute it according to its owners restrictions, if any.

> Some of us aren't OARC members, and likely don't qualify to be.

I'm not sure what you mean by "don't qualify": OARC membership is open
to DNS operators, implementers and researchers.  I think anyone
interested enough in DNS to read namedroppers could make a compelling
case for membership.  While it would be ideal that if you support
OARC's mission, you'd choose a paying membership category, it's also
possible to apply as a beneficial member (which requires a yearly
material, non-financial contribution, such as contributing data or
presenting research).

More information about OARC is available at www.dns-oarc.net or please
feel free to contact me privately.

Matt

---

From: David Conrad <[email-masked]>
Date: Thu, 20 Aug 2009 17:55:42 -1000

On Aug 20, 2009, at 4:02 AM, Andrew Sullivan wrote:
> disagree.  As usual, this all hinges on the value of "works".

I believe the controversy is less with "it works" than with "it  
violates RFC 3226".  It seems pretty clear to me that any normal  
reading of that RFC would indicate that BIND is in violation of 3226.   
The question is whether that matters.

Right now, in an analysis done for the "L" root server replaying  
modified live data, we see a large jump in TCP queries largely due to  
the "fall back to EDNS@512" BINDism, but the number of TCP queries is  
still pretty much in the noise and well within the tolerances of  
"L" (can't speak to other root servers of course).  However let's  
posit an alternative reality in which people actually turn on  
validation.  A root DNSKEY response is 1749 bytes.  What does this  
imply?

Thanks,
-drc

---

From: Paul Vixie <[email-masked]>
Date: Fri, 21 Aug 2009 06:34:58 +0000

> From: David Conrad <[email-masked]>
> Date: Thu, 20 Aug 2009 17:55:42 -1000
> 
> I believe the controversy is less with "it works" than with "it violates
> RFC 3226".  It seems pretty clear to me that any normal reading of that
> RFC would indicate that BIND is in violation of 3226.  The question is
> whether that matters.

speaking for RFC 2671, if EDNS0 @4096 does not work then EDNS0 should be
negatively cached for that server, which means use DNS if it can carry the
response data you need (which is DNSSEC metadata in this case, so, "don't").
fallback to 1500, 1480, 1420, 1400, 1280, 1024, or especially 512, is too
much initiator-side state, too much complexity, too much application delay,
too much DNS traffic, and too much namedroppers traffic.  so, we should not
do it.  (note, i'm not speaking as president of ISC here.)

speaking for RFC 3226, the DO bit is being treated as by some as "DNSSEC
would be understood so it's safe to send it" and by others as "DNSSEC is
wanted so please do send it."  i prefer the former interpretation since it
is minimally specificative.  (is that a word?)  the former interpretation
would allow a responder to say, if the buffer size isn't large enough to
carry DNSSEC metadata, then pretend DO=0.  the latter interpretation would
require an initator to say, if i really do want DNSSEC metadata than i need
a buffer size of 1220 or more, so i should not set bufsize=512 and DO=1.

> However let's posit an alternative reality in which people actually turn
> on validation.  A root DNSKEY response is 1749 bytes.  What does this
> imply?

yipe!

---

From: Doug Barton <[email-masked]>
Date: Sat, 22 Aug 2009 21:54:31 -0700

> to 6000 UDP requests/second.

Whose traffic? I assume we're talking about ORG? Where is this documented?

Meanwhile, we're losing sleep because 1% of connections/second are now
happening over TCP? Has anyone from Afilias actually said that there
is a problem here? Do any other sites anticipate a problem if 1% (or
even *gasp* 2%) of their connections are TCP?

I've already stated my support for wanting to see something "better"
than we have now for making sure that EDNS is going to work, and I'm
sorry if I'm being dense but I don't see that this problem justifies
the amount of drama we've seen on the list so far.

Doug (as always hoping to be enlightened if I'm wrong)