Subject: [dnsext] EDNS0 fallback "revisited"

From: Olafur Gudmundsson/DNSEXT chair <[email-masked]>
Date: Tue, 25 Aug 2009 13:44:28 -0400

Dear colleagues,

After reading all the messages on EDNS0 buffer sizes and fallback
your chairs have huddled and tried to agree on an approach to go forward
and what/if protocol work is needed.

For the sake of argument we want to limit the discussion to
         "ENDS0 capable entities"
and their behavior.  The goal is to minimize fallback.
By decreasing fallback cases we speed up DNS resolution for everyone.

We think that the current evidence suggests three main problems:
     - inability or refusal to handle UDP packets larger than 600 bytes.
     - inability or refusal to handle UDP fragments.
     - failure to set the advertised EDNS0 buffer sizes correctly given
       the actual network conditions

Model:
A is the issuer of a DNS query
B is the target server of the query

         A <---pipe1---> INTERNET <----pipe2----> B

If A advertises a ENDS0 buffer size that is larger than will go through
"pipe1" then B may send back an answer that is larger than fits through the
pipe. Then A will time-out and do a fallback.

If B attempts to put an answer into "pipe2" that is larger than fits into
it A will not get an answer and will do a fallback.

If both A and B know the "width" of their LOCAL pipes fallback
between them is rare and if B has an answer that does not fit, it will
set the TC bit.

If the problem is actually in the INTERNET part of the diagram,
fallback is needed no matter what.  We can put of evaluation of how
much of the problem is in fact in the middle, because we know for sure
right now that there are problems at the end points.  Since those are
under the control of the affected parties, we can tackle this problem
first.

The assumption here is A and B can detect the "width" of their local pipes
and either fix them or set UDP-buffersize parameters to values that their
local pipe will pass. If either A or B have multiple paths to the Internet
they should set their buffer sizes to the smallest pipe width.
In particular B should NEVER provide an answer to a normal (i.e. non ANY and
non RRSIG) query that does not pass through "Pipe2".

Work to be done:
-  Provide tools that measure local pipes and provide configuration hints
-  Educate users to use the tools, where to look for problems and how
         to update configurations.
-  Make the firewall community aware that firewall's are causing DNS problems
    and educate them in how to update their configurations, to allow well
    formed UDP fragments to pass. (i.e. non overlapping fragments)
-  ditto for DNS proxy vendors.
-  Improve EDNS0 fallback strategies.

Only the last one is a protocol work that DNSEXT can undertake, all the others
are external educations activities that are outside the scope of the working
group.

As DNS is not the only UDP based protocol that is affected by the 
restrictions on
UDP packets, modifying DNS to work around misbehaving middle boxes is counter
productive to making the Internet better in the long run.
Adding complexity to DNS to handle the misbehaving units is a wasted effort
as the educational tasks will still have to be undertaken and provide a
quicker return on investment.

         Olafur & Andrew

---

From: Alex Bligh <[email-masked]>
Date: Tue, 25 Aug 2009 19:32:31 +0100

--On 25 August 2009 13:44:28 -0400 "Olafur Gudmundsson/DNSEXT chair" 
<[email-masked]> wrote:

> Only the last one is a protocol work that DNSEXT can undertake, all the
> others are external educations activities that are outside the scope
> of the working group.

a) Re 2nd & 3rd, was BCP 152 / RFC 5625 out of scope for the working group?

b) I note that as currently formulated, 4th would exclude (e.g.) EDNS Paging
   strategies if not implemented as a fallback). Is this deliberate?
   It might be that the best strategy when querying for DNSSEC records is
   to use as a first strategy (rather than a fallback) a technique that
   that will allow replies greater than 1492 bytes without
   fragmentation, given large replies seem common and there seems to be
   at some degree of support for the idea that either fragmentation is
   bad and should be avoided per se (even if middleboxen were perfectly
   behaved), or that it is commonly dropped by middleboxen. Perhaps your
   final para (which I chopped off - sorry) precludes this approach.

c) I would have thought new algorithms to reduce the prevalence of large
   replies (e.g. ECC) would be in scope for this w/g. This is arguably
   desirable quite aside from the middlebox problem.

-- 
Alex Bligh

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 25 Aug 2009 12:31:08 -0700

>  final para (which I chopped off - sorry) precludes this approach.

Its actually interesting.  I'll argue as follows:

Fragmentation is NOT a problem for ~85% +/- some delta of resolvers.   
Which means, for MOST resolvers, its OK if the response fragments.  So  
let the responses be large, we have these large DNSSEC responses  
anyway, so go for it.

But fragmentation IS a problem for ~15%.

This is probably too large a number for education strategies to work,  
since that requires upgrading/replacing existing middleboxes.

And too large a number for any strategy which requires timeouts per  
authority before deciding its a problem, which will kill performance  
and cause "DNSSEC is broken" complaints once .com starts signing data.

But it is a small enough number that these CAN specify an EDNS buffer  
small enough to fragment, if the resolver can detect that its a global  
problem, with fallback to TCP on truncation: you're talking ~10-20% of  
the total queries, assuming all responses exceed the ~1400 MTU of a  
typical fallback: significant but hardly a killer.

Especially since the .org TCP load is probably representative:  
significant but should be tolerable.  Resolvers behind broken  
middleboxes will today do the fallback to TCP eventually when  
querying .org for a large response, they are just going through an  
annoying, performance-killing timeout first.

As for authorities, if the authority has a problem SENDING fragments,  
let it get hammered by having all its responses over TCP:  Either it  
can keep up with the load and they only see an extra couple of RTTs on  
DNS lookups, or it falls over dead until the offending middlebox get  
fixed: a self-correcting situation.

And as for the rest of the Internet protocols, I think its probably  
safe to say everyone else has given up on UDP fragments working always  
on the Internet, judging by the workarounds for IKE.  Thus I don't see  
the particular problem in the DNS protocol saying "We MUST NOT rely on  
UDP fragmentation working in all cases, but MAY rely on fragmentation  
working in most cases"

Therefore, I believe the objective should be "Make sure that DNSSEC  
works on an Internet that MOSTLY is OK with UDP fragments, but  
understands that some resolvers can't handle fragments, and some  
authorities can't handle fragments", without ANY protocol changes and  
single-sided changes to just resolvers.  Which appears quite doable.

---

From: Paul Vixie <[email-masked]>
Date: Tue, 25 Aug 2009 20:42:39 +0000

> From: Nicholas Weaver <[email-masked]>
> Date: Tue, 25 Aug 2009 12:31:08 -0700
> ...
> And as for the rest of the Internet protocols, I think its probably safe
> to say everyone else has given up on UDP fragments working always on the
> Internet, judging by the workarounds for IKE.  Thus I don't see the
> particular problem in the DNS protocol saying "We MUST NOT rely on UDP
> fragmentation working in all cases, but MAY rely on fragmentation working
> in most cases"
> 
> Therefore, I believe the objective should be "Make sure that DNSSEC works
> on an Internet that MOSTLY is OK with UDP fragments, but understands that
> some resolvers can't handle fragments, and some authorities can't handle
> fragments", without ANY protocol changes and single-sided changes to just
> resolvers.  Which appears quite doable.

i recognize in the above text the voice of reason, and i concur with it.

meanwhile i still hope for improvements to tcp itself or udp itself, and
i have not quite given up on sctp.

---

From: bert hubert <[email-masked]>
Date: Wed, 26 Aug 2009 00:22:46 +0200

On Tue, Aug 25, 2009 at 9:31 PM, Nicholas
Weaver<[email-masked]> wrote:
> Fragmentation is NOT a problem for ~85% +/- some delta of resolvers.  Which
> means, for MOST resolvers, its OK if the response fragments.  So let the
> responses be large, we have these large DNSSEC responses anyway, so go for
> it.

Nicholas,

In order to guide this discussion, it would help tremendously if you
had some report of all your relevant DNS/UDP/TCP statistics (with
their caveats). Have you published something like that? Would you
consider doing so?

Because that would be great.

Thanks.

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 25 Aug 2009 15:44:41 -0700

> Because that would be great.

We didn't do anything with TCP: in fact, our authority is UDP only for  
testing.

The stats for fragmentation behavior based on EDNS size we already  
posted in the group, although we want to do an extended version that  
checks reordering and similar, but thats on the low-priortiy todo pile  
because we need to write up everything else.

---

From: Eric Osterweil <[email-masked]>
Date: Mon, 31 Aug 2009 14:32:44 -0600

Sorry for the Johnny-come-lately response:

> -  Provide tools that measure local pipes and provide configuration  

We presented dnsfunnel for this at IETF 75, and I think it would be  
great for people to download it and try it out to see this for  
themselves:
	http://www.vantage-points.org/

You can point dnsfunnel at any zone and it will tell you if there is a  
problem and what sizes fit.  This has the benefit of testing the path  
from _your_ stub to each name server in whichever zone you aim it at.

Eric
é

---

From: Eric Osterweil <[email-masked]>
Date: Mon, 31 Aug 2009 14:32:44 -0600

Eric
é

---

From: Eric Osterweil <[email-masked]>
Date: Mon, 31 Aug 2009 14:35:56 -0600

On Aug 25, 2009, at 1:31 PM, Nicholas Weaver wrote:

>> --On 25 August 2009 13:44:28 -0400 "Olafur Gudmundsson/DNSEXT  
>> chair" <[email-masked]> wrote:
>>
>>> Work to be done:
> Fragmentation is NOT a problem for ~85% +/- some delta of  
> resolvers.  Which means, for MOST resolvers, its OK if the response  
> fragments.  So let the responses be large, we have these large  
> DNSSEC responses anyway, so go for it.
>
>
>
> This is probably too large a number for education strategies to  
> work, since that requires upgrading/replacing existing middleboxes.
>
> But it is a small enough number that these CAN specify an EDNS  
> buffer small enough to fragment, if the resolver can detect that its  
> a global problem, with fallback to TCP on truncation: you're talking  
> ~10-20% of the total queries, assuming all responses exceed the  
> ~1400 MTU of a typical fallback: significant but hardly a killer.

First, a caveat: this is a position I completely agree with (as I  
think I've advocated it since the very beginning, back in 2008).

However, I think this discussion needs to keep in mind that this is a  
moving target.  Key-set sizes change as keys and revocations, etc are  
added and removed.  Moreover, the path characteristics change too.   
This is why SecSpider is doing a longitudinal study.  If we say 85%  
today, it does not necessarily mean this will be the right # tomorrow.

Ultimately, I think a the protocol should expose semantics that let  
client code adapt to these PMTU problems as it encounters them. This  
should let resolvers handle problems without any operator intervention.

> As for authorities, if the authority has a problem SENDING  
> fragments, let it get hammered by having all its responses over  
> TCP:  Either it can keep up with the load and they only see an extra  
> couple of RTTs on DNS lookups, or it falls over dead until the  
> offending middlebox get fixed: a self-correcting situation.
>
>
> safe to say everyone else has given up on UDP fragments working  
> always on the Internet, judging by the workarounds for IKE.  Thus I  
> don't see the particular problem in the DNS protocol saying "We MUST  
> NOT rely on UDP fragmentation working in all cases, but MAY rely on  
> fragmentation working in most cases"
>
> authorities can't handle fragments", without ANY protocol changes  
> and single-sided changes to just resolvers.  Which appears quite  
> doable.

How about, make DNSSEC avoid falling back to fragmented UDP packets  
when it can avoid this by detecting PMTU limits?  THen there is no  
declaration about the brokenness of anything, simply a statement of  
how to proceed?

Eric
é

---

From: Eric Osterweil <[email-masked]>
Date: Mon, 31 Aug 2009 14:35:56 -0600

Eric
é

---

From: Eric Osterweil <[email-masked]>
Date: Mon, 31 Aug 2009 15:05:10 -0600

Eric
é

---

From: Patrik Fältström <[email-masked]>
Date: Tue, 1 Sep 2009 07:21:51 +0200

> But fragmentation IS a problem for ~15%.

It is a bit more complicated than this. It might be ok with  
fragmentation, as long as the fragments arrive in order. This implies  
it even might work _sometimes_.

> work, since that requires upgrading/replacing existing middleboxes.

Well, the 85% number is high enough so that we can say "this is how it  
should be"...but in general I agree with you.

> and cause "DNSSEC is broken" complaints once .com starts signing data.

The question is who the complaint is going to. Some DNSSEC problems  
will target the large ISPs that run the large resolvers. They will  
detect people with broken signed zones -- but they will most certainly  
_not_ have the fragmentation issues (or they have serious problems  
anyway). The complaints on fragmentation will hit the smaller  
resolvers that sit behind firewalls that do the wrong thing. I presume  
here people have correct firewalls in front of authoritative servers  
with signed zones -- because here I also think we have other problems  
if people have that broken stuff.

I am optimistic here. If something causes a delay (the time for a  
fallback to TCP), then people will start to investigate why that  
happens. In a way similar to the IPv6 deployment issue. Why do we get  
a timeout? Aha, the IPv6 routing was broken, and we have to fall back  
to IPv4.

> significant but should be tolerable.
>  Resolvers behind broken middleboxes will today do the fallback to  
> TCP eventually when querying .org for a large response, they are  
> just going through an annoying, performance-killing timeout first.

...and then fix their middlebox...or send correct EDNS0 bufsize...or...

> and single-sided changes to just resolvers.  Which appears quite  

Maybe tune this a bit. It might not be the resolver or authorities  
that can not handle fragments. It might be something in between.

Otherwise I am with this.

    Patrik