Subject: [dnsext] Response to request for real data...

From: Nicholas Weaver <[email-masked]>
Date: Mon, 29 Jun 2009 10:37:49 -0700

A preliminary analysis of EDNS behavior

This represents some preliminary analysis from Netalyzr (http://netalyzr.icsi.berkeley.edu 
) with how EDNS is used and supportable by both end-user connections  
and by recursive resolvers.  To date, over 40,000 distinct IPs have  
run Netalyzr to completion.

Please note that there is bias in terms of the users who have run  
Netalyzr, arising from how users learn about our site and what leads  
them to decide to run the tests.  In particular, we already know that  
an unusually high fraction of IPs are from Comcast (~4900) due to an  
initial slashdot article which mentioned Netalyzr in the context of  
assessing Comcast, and a large number are from t-dialin.net (~1500,  
tied for third in popularity) due to German-language coverage.   
Another likely bias is the popularity of OpenDNS: over 11% of  
Netalyzr's users appear to use OpenDNS as their recursive resolver.

Likewise, there are double-counting issues when viewing data by  
resolver, and packet loss and timeouts may cause some figures to read  
high or low, adding noise to the results.

Additionally, Netalyzr's test suite with regard to EDNS has not been  
static: the tests to determine the root cause of large-payload EDNS  
failures was a recent addition.  Thus, for some of these tests there  
are far fewer data points available.

		       Description of the tests

Netalyzr works by having a Java applet on the user's computer conduct  
tests with servers we control.  In particular, the node providing the  
applet hosts a DNS server, and this DNS server code is also running as  
the authority for .netalyzr.icsi.berkeley.edu.

The DNS server is a custom Python implementation, and produces both  
standard and deliberately nonstandard responses for various queries.

To determine if a user's resolver is EDNS or DNSSEC capable, the  
addresses has_edns.{x}.netalyzr.icsi.berkeley.edu and wants_dnssec...  
are queried by the Java applet.   These return "true" if the condition  
is true, and false otherwise. (True is encoded by returning the IP of  
the back-end node the user is connected to, false uses a different IP  
address.)  The EDNS0 OPT record, if present, is returned unchanged in  
the response.

To determine the user's resolver's identity, the applet queries for  
server.{x}, which returns the IP address that our DNS authority  
received the query from.  Note that this may cause a single resolver  
to be associated with multiple identities (creating double-counting)  
if that resolver generates requests through multiple IP addresses.

To determine the user's resolver's EDNS MTU, the query edns_mtu...  
encodes the EDNS MTU in the two lower octets of the returned A  
record.  This test is only run if the resolver indicates support for  
EDNS, and can only be captured if the user trusts the applet.

To determine if the user's resolver can receive a large EDNS response,  
the applet queries for edns_large.{x}.  This returns the true IP, but  
has a large number of CNAMES in the additional field, which is  
sufficient to pad the packet to ~1800B.  This response is fragmented  
by the server when it is sent, as this is greater than the Ethernet  
MTU.  Since the CNAMES are effectively garbage, we expect that most  
recursive resolvers will not return these as part of the final  
answer.  This test is only run if the resolver indicates support for  
EDNS.

To determine if the user's resolver can receive a medium sized EDNS  
response, the applet queries for edns_medium.{x}. This returns the  
true IP, but has a large number of CNAMES in the additional field,  
which is sufficient to pad the packet to ~1400B.  This response is NOT  
fragmented by the server when it is sent, as this is less than the  
Ethernet MTU.  It may be fragmented in the network later.  This test  
is only run if the resolver indicates support for EDNS.

To prevent ordering and history issues, the first tests are for EDNS  
and DO support, then EDNS MTU, then for medium-sized EDNS responses  
and then large EDNS responses.  Thus, resolvers that maintain state in  
determining EDNS0 MTU and having a malfunctioning middlebox should, at  
least for the first run, not use a smaller than default EDNS0 MTU.

Finally, the applet does a direct UDP-based DNS request (it tries 5  
times until successful) directly to our server to query for  
edns_large, edns_medium, and edns_small (an EDNS-present response of  
~300B), to check for the direct behavior of any DNS-aware  
middleboxes.  These tests do not validate the response as correct, but  
other tests check for DNS-modifying middleboxes.

		       EDNS and DNSSEC support

If we view based on user IP address, 49% (19968 out of 40476) IP  
addresses use a recursive resolver that used EDNS0 in requests to our  
authority.  46% (18634 out of 40476) have DO enabled, indicating they  
wish to receive DNSSEC records.

Please note a known bias: Comcast's resolvers, although they support  
EDNS0 when tested manually, do not appear to use it by default in most  
requests to our authority.  As this represents 12% of all IP addresses  
which have run Netalyzr, this is a potentially significant undercount.  
Thus, the actual fraction of users behind an EDNS0 and/or DNSSEC using  
resolver may be somewhat higher.

If we instead view based on observed resolver IP, this check covers  
10852 recursive resolvers external addresses.  Of these, 65% (7074 out  
of 10852) have EDNS0 enabled and a MTU > 1200B, with 58% (6286 out of  
10852) having DO set as well.

Please note a known bias: If a recursive resolver uses multiple IP  
addresses to generate queries, we will count it multiple times.  For  
example, this counting method counts OpenDNS as at least 33 separate  
resolvers.

	       EDNS failures and their probable causes

If we view based on IP, 18458 IPs both have a resolver which  
advertises EDNS support and where we captured an advertised EDNS MTU  
greater than 1900B.  Of these, only 87% (16050 out of 18458)  
successfully received the EDNS-large response.

The more sophisticated version, which checks both large and medium,  
has far fewer data points.  Of these, 87% (1946 out of 2232) were able  
to receive the large EDNS response, while 97.2% (2170 out of 2232)  
were able to receive the medium sized EDNS response.

Likewise, if we view based on resolver, 86% (1284 out of 1489) were  
able to receive the large response and 96% (1431 out of 1489) were  
able to receive the medium response.

Systems which can't handle large EDNS responses but can handle the  
medium sized responses are likely due to a middlebox which either  
can't handle DNS fragments, fragmented UDP traffic, or which assumes  
that DNS responses have to be smaller than ~1500B.

Systems that can't handle the large and medium EDNS responses are  
likely due to middleboxes which can't handle DNS responses > 512B.

If the middlebox couldn't handle EDNS, the edns-present query would  
have failed.

The situation is similar for the end user's connection.  On a query  
directly to our server from the applet, bypassing the recursive  
resolver, the EDNS large request is successful 86% of the time (3635  
out of 4211), the medium is successful 95% of the time (4019 out of  
4211), and the small is successful 99% of the time (4179 out of 4211).

This suggests that for the end user there is a significant problem  
with middleboxes.  Many middleboxes either can't NAT UDP fragments or  
do not reassemble fragmented DNS traffic before analysis, but a  
significant number simply assume that DNS packets are smaller than  
1400B.

We did not test the ability for middleboxes to parse DNSSEC responses  
or unknown RRTypes.  This could also be a concern with DNSSEC  
deployments.

			     Conclusions

The natural conclusion is there are significant numbers of "DNS aware"  
middleboxes in the paths for both recursive resolvers and end-users  
that are incorrect: they fail to properly pass large DNS responses. Of  
the failures, the most common failure is that they don't reassemble  
fragments before processing DNS traffic, (or have some other failure
which prevents > 1500B DNS responses from being processed), with an  
inability to handle > 512B responses a second concern.

Among other things, this suggests that DNSSEC-using authorities that  
wish to minimize EDNS failures and/or TCP fallback attempt to,  
whenever possible, ensure that responses fit within the ethernet MTU.

--
to unsubscribe send a message to [email-masked] with
the word 'unsubscribe' in a single line as the message text body.
archive: <http://ops.ietf.org/lists/namedroppers/>