Subject: [dane] Large TLSA RRs (X 0 0) meet reality?

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 21 Jun 2013 14:50:46 +0000

[ Bcc'd to domain owner of large "2 0 0" TLSA RR. ]

In testing various SMTP server's TLSA RRs I found that at least
one prominent validating nameserver drops requests for large TLSA
RRs over UDP.  (The TLSA RR is returned if the request does not
set the "DO" bit).

With no UDP response at all (not even TC=1) clients don't fall back
to TCP.  In particular this is observed with Google's 8.8.8.8 DNS
service.  Queries for a "2 0 0" TLSA RR containing an 1885 byte
certificate time out over UDP.  And even with TCP forced, the
response is returned with the "AD" bit incorrectly not set.

    $ drill -D -t _25._tcp.mx.<censored>.org @8.8.8.8 tlsa
    ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 48420
    ;; flags: qr rd ra ; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
    ;; QUESTION SECTION:
    ;; _25._tcp.mx.<censored>.org.    IN      TLSA

    ;; ANSWER SECTION:
    _25._tcp.mx.<censored>.org.       0       IN      CNAME _cacert-c3-tlsa.<censored>.org.
    _25._tcp.mx.<censored>.org.       0       IN      RRSIG   CNAME 7 5 7200 20130630103058 20130531094020 56225 <censored>.org. ...
    _cacert-c3-tlsa.<censored>.org.   0       IN      TLSA    2 0 0 <1885 bytes>
    _cacert-c3-tlsa.<censored>.org.   0       IN      RRSIG   TLSA 7 3 7200 20130630103058 20130531094020 56225 <censored>.org.  ...

    ;; Query time: 376 msec
    ;; EDNS: version 0; flags: do ; udp: 512
    ;; SERVER: 8.8.8.8
    ;; WHEN: Fri Jun 21 09:57:52 2013
    ;; MSG SIZE  rcvd: 2327

Queries for more modestly sized "X Y 1", RRs are handled just fine.

    $ drill -D _25._tcp.master.debian.org. tlsa @8.8.8.8
    ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 1645
    ;; flags: qr rd ra ad ; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0
    ;; QUESTION SECTION:
    ;; _25._tcp.master.debian.org.  IN      TLSA

    ;; ANSWER SECTION:
    _25._tcp.master.debian.org.     3600    IN      TLSA    3 1 1 b5fcc059553aafb1ba54294208479c154d940d9b36c1a22ca41bd1c041c87b26
    _25._tcp.master.debian.org.     3600    IN      RRSIG   TLSA 7 5 3600 20130718215759 20130620215759 17309 debian.org. cIP8ZqPkITt6flZu2qMXXzZmiHC07SdnGiEtq+FiPlkvhDpx3CZPfQVMboaFDkR5vp4Ql9eMBIBr4N0GoGIfPPI2oV5uYNjjdmRoV9XAsRgBKn9bTCAaiumBOMjWA9+u3/WND9r2/qB13M2j6JAiU/QDSZky7qs5icqhaezjnR7NPcy+89Zk6yS2lU7Qg079

    ;; Query time: 400 msec
    ;; EDNS: version 0; flags: do ; udp: 512
    ;; SERVER: 8.8.8.8
    ;; WHEN: Fri Jun 21 10:33:53 2013
    ;; MSG SIZE  rcvd: 288

Though of course one should typically not rely on the "AD" bit from
8.8.8.8 for lack of transport security between the client and the
nameserver, this is in indicator of likely trouble.  So I think my
advice against "X 0 0" stands, these RRs are too large to be reliably
used with DNS.

-- 
	Viktor.

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 21 Jun 2013 10:57:04 -0400

On Fri, Jun 21, 2013 at 02:50:46PM +0000, Viktor Dukhovni wrote:

> certificate time out over UDP. 

Are you quite sure that your problem isn't that you're sending your
UDP query with an EDNS0 buffer size larger than you can actually
receive?  That's often the problem here.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 21 Jun 2013 16:34:10 +0000

> receive?  That's often the problem here.

Fairly confident, I've tested two different platforms at two
different sites.  I'll send you the domain off list, please verify
for yourself. [ I don't want to publish the domain withour the owner's
permission. ]

-- 
	Viktor.

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 21 Jun 2013 13:47:19 -0400

> for yourself. [ I don't want to publish the domain withour the owner's

Thanks.  I tried this.

If I use -b up to about 2300, I get back a truncated answer (so I'd
retry over TCP).  If I go above that, however, it hangs.  This makes
sense; I'm pretty sure my network is going to fragment at that point,
and lots of people don't reassemble UDP fragments.  

So I think the problem here is that you can't effectively query such a
large answer in most networks over UDP, because the EDNS0 buffer size
you'd need for the answer to fit is too large for most networks.  I'm
not sure that's evidence that the particular use is bad.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Richard Barnes <[email-masked]>
Date: Fri, 21 Jun 2013 14:28:53 -0400

So is the answer here just, "Use TCP"?  Like for DNSSEC records?

On Fri, Jun 21, 2013 at 1:47 PM, Andrew Sullivan <[email-masked]>wrote:

> _______________________________________________
> dane mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dane
>

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 21 Jun 2013 14:43:38 -0400

You don't have to use TCP for DNSSEC either, but sometimes you do.  It
seems in this case you need to.

-- 
Andrew Sullivan
[email-masked]

---

From: Viktor Dukhovni <[email-masked]>
Date: Fri, 21 Jun 2013 21:27:55 +0000

> and lots of people don't reassemble UDP fragments.  

I don't know why, they would not.  It is a pretty basic requirement.
The Ethernet IP MTU is <= 1500, so fragment should set in well
before 2300 bytes.  The problem sets in when the answer is not
truncated.  Note the TCP response was  

    ;; SERVER: 8.8.8.8
    ;; MSG SIZE  rcvd: 2327

So the 2300 is not about cross the UDP re-assembly limit, but rather
crossing the point at which the entire response fits untrucated.

> not sure that's evidence that the particular use is bad.

I bet it is something much more interesting.  In any case the point
is that it fails pretty badly, hence such RRs are unwise.

-- 
	Viktor.

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 21 Jun 2013 18:12:05 -0400

> I don't know why, they would not.  It is a pretty basic requirement.

I've now endured two presentations at IETF meetings in which large
operators have said, "We don't reassemble UDP fragments; fragments
suck."  

> before 2300 bytes.

Yes, but in this case it doesn't matter.

>  The problem sets in when the answer is not

I clearly didn't explain myself well enough.  If you have an EDNS0
bufsize below 2327 bytes, then the answer never fits.  Therefore, the
server simply responds with a TC bit and no answer.

If, however, you say that your bufsize is over 2326, then the server
attempts to answer you sending a UDP packet with the necessary buffer
size.  This doesn't work, because your buffer isn't actually that big,
and therefore the whole thing hangs.  _That's_ the problem: when you
sent the query, your buffer size was too big.  Yes, this is a problem,
but real resolvers are learning not to work the way drill does: they
work to figure out a safe bufsize and then stick to it.  (Drill
doesn't do this because it's a diagnostic tool.  It should do what you
tell it.)

> I bet it is something much more interesting. 

Having just performed the exercise that makes my point, I'll
cheerfully take that bet!

> In any case the point

I don't believe your illustration using drill shows what you think it
does.  A properly-behaved resolver shouldn't fail, but should get the
TC-bit response and fall back to TCP.

There's a live question as to whether falling back to TCP is
acceptable; I think the operator of the service gets to make that
determination, however.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 22 Jun 2013 04:17:35 +0000

> operators have said, "We don't reassemble UDP fragments; fragments

This is irrelevant, IP reassembly happens at the receiving endpoint
(my machine) not in the network core (but sometimes also in NAT
devices and firewalls near the sending or receiving system).  So
whatever large operators are anecdotally reported to do is of little
consequence.

> server simply responds with a TC bit and no answer.

This is not always how the TC bit works, in many cases as much of
the answer is sent as fits in the sender's buffer, for example with
a query buffer size of 2200 sent to the authoritative server of
the domain in question, I get a truncated response sent in two UDP
fragments:

    23:19:44.535213 IP (tos 0x0, ttl 63, id 32246, offset 0, flags [none], proto
    UDP (17), length 81)
	192.0.2.1.59702 > 192.0.2.2.53: [udp sum ok] 14849+ [1au]
    Type52? _25._tcp.mx.censored.org. ar: . OPT UDPsize=2200 OK (53)

    23:19:44.625789 IP (tos 0x0, ttl 45, id 29143, offset 0, flags [+], proto
    UDP (17), length 1396)
	192.0.2.2.53 > 192.0.2.1.59702: 14849*-| q: Type52?
    _25._tcp.mx.censored.org. 3/0/1 _25._tcp.mx.censored.org. [2h] CNAME _cacert-c3-tlsa.censored.org.,
    _25._tcp.mx.censored.org. [2h] RRSIG, _cacert-c3-tlsa.censored.org. [2h] Type52[|domain]

    23:19:44.625822 IP (tos 0x0, ttl 45, id 29143, offset 1376, flags [none],
    proto UDP (17), length 807)
	192.0.2.2 > 192.0.2.1: udp

showing clearly that in this case TC is sent with a truncated
answer.  With Google's 8.8.8.8 however, truncation happens sooner
and since the answer RRset includes just one RR, indeed truncation
sends a TC bit with zero answers.

> and therefore the whole thing hangs.

My buffer is that big, so if there is a re-assembly issue it could
only be in the load-balancer or firewall/NAT hardware handling
8.8.8.8, since core routers don't do or care about reassembly, they
just switch IP datagrams.

>  _That's_ the problem: when you

I tested with a "real resolver", the MacOSX 10.8.3 system resolver
library is reasonably recent.  When I set the nameserver to 8.8.8.8,
res_search() DNS lookups time out.

    # networksetup -setdnsservers Wi-Fi 8.8.8.8

    $ posttls-finger -t 30 -T 180 censored.org
    posttls-finger: warning: DANE TLSA lookup problem: Host or domain name not found. Name service error for name=_25._tcp.mx.censored.org type=TLSA: Host not found, try again
    posttls-finger: Failed to establish session to censored.org via mx.censored.org: TLSA lookup error for mx.censored.org:25

Using my OpenWRT router:

    # networksetup -setdnsservers Wi-Fi 192.168.1.1

    $ posttls-finger -t 30 -T 180 censored.org
    posttls-finger: using DANE RR: _25._tcp.mx.censored.org -> _cacert-c3-tlsa.censored.org IN TLSA 2 0 0 <certificate>; sha256 digest ...
    posttls-finger: Connected to mx.censored.org[192.0.2.1]:25
    posttls-finger: mx.censored.org[94.142.241.89]:25: depth=0 chain is trust-anchor signed
    posttls-finger: Verified TLS connection established to mx.censored.org[94.142.241.89]:25: TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)

> TC-bit response and fall back to TCP.

If system resolver libraries in MacOSX and NetBSD are not properly-
behaved, my point still stands.  The bottom line is that large
DNS payloads are prone to timeout with no client recovery via TCP.

> acceptable; I think the operator of the service gets to make that

Not possible unless an answer comes back with the TC bit.  The
client initiates TCP retries, the server has no TCP listening port
on the client to connect back to.

Regardless of the details of the failure mode, the point is that
it fails with real tests (I was testing the Postfix SMTP stack,
and only used "drill" later to show the behaviour I found).  Thus
"X 0 0" with large certificates is not robust.

If various DNS geo load balancers can't handle large UDP responses,
publishing large DNS payloads is going to be a problem.  The operational
consequence for DANE is "shun IN TLSA X 0 0, it is fragile".

-- 
	Viktor.

---

From: Viktor Dukhovni <[email-masked]>
Date: Sat, 22 Jun 2013 21:49:45 +0000

> consequence for DANE is "shun IN TLSA X 0 0, it is fragile".

So as not to appear to be engaging in a point-by-point DNS-internals
slugfest with Andrew (something neither I nor likely he have any
interest in), I just want to return to the original point I was
trying to make.  Administrators of TLS servers should probably
avoid "X 0 0" TLSA records, they are liable to not work (SRVFAIL)
with various firewalls, load-balancers, ... at either the server
or the resolver end.

This does not mean that clients should not implement this combination,
after all if the packets get through, by all means support the RR.
Rather, server operators should not publish them for operational
reliability reasons.  So in practice I expect to see minimal
deployment of these TLSA RR forms.

-- 
	Viktor.

---

From: Mark Andrews <[email-masked]>
Date: Mon, 24 Jun 2013 09:07:14 +1000

> deployment of these TLSA RR forms.

Or we could just make fragmented UDP work.  Add a hop-by-hop option
that contains the UDP ports then firewalls can be selective about
which fragments they let thrrough.  Fragments themselves are not
the issue most of the time.  It is ensuring the reply packet is
part of a flow and the necessary information to do this is not in
all of the fragments.  One could do this with 8 octets per fragment
for UDP fragments.

DNS clients shouldn't have to put up with a DoS attack on them by
the firewall.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 24 Jun 2013 03:33:47 +0000

> DNS clients shouldn't have to put up with a DoS attack on them by

I agree that UDP fragments should work, but I can't honestly say
that this is an alternative to recommending that administrators
not deploy "IN TLSA X 0 0" certificate associations.  Devices that
don't support UDP fragments are going to be with us for a long
time, much longer than the 5 year horizon for DNSSEC and DANE to
begin to become mainstream (or else proved unusable and forgotten).

Sadly we have an Internet that evolved, not an Internet that might
have been designed with 20-20 hindsight.

-- 
	Viktor.

---

From: Mark Andrews <[email-masked]>
Date: Mon, 24 Jun 2013 15:25:33 +1000

In message <[email-masked]>, Viktor Dukhovni writ
> have been designed with 20-20 hindsight.

Stub resolvers just need to advertise UDP buffer sizes that work for
the firewalls they deploy.  Extend DHCP/RA nameserver information to
have a edns udp size.  Those behind broken firewalls can set to
a small value.  Those that are not behind broken firewalls can set
it to a sane value.

We could add EDNS rcode that firewalls could return to say "EDNS
UDP too big" along with a working EDNS size.  Firewalls would
intercept the query and return a reply to it with this rcode set.
Firewalls could participate in the protocol instead of destroying
it.

Mark

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 24 Jun 2013 14:09:40 +0000

> Firewalls could participate in the protocol instead of destroying

The above still looks like wishful thinking.  Sure the problems
could be solved "in principle", but I see little chance of a solution
in practice.

The problem I observed was at the DNS nameserver end.  My OpenWRT
NAT router handles UDP fragments just fine.  [ It also runs unbound
and handles DNSSEC, so I don't normally use 8.8.8.8 as my nameserver,
but the experiment was I believe worthwhile. ]

I am afraid I see no alternatives to keeping DNS RRset sizes
conservatively small.

What could help is, and is a good idea to have on every machine is
a local validating caching nameserver on 127.0.0.1:53 on every
computer (even mobile devices).

Unlike applications using limited functionality stub resolvers, a
local cache can quickly learn whether its upstream forwarders
support EDNS0 and what EDNS0 buffer sizes are viable.  This should
be done in a way that does not rely on any new features in network
equipment between the device and the forwarder.

And yet, it will still be prudent to not publish full certificates
in DNS.

-- 
	Viktor.