Subject: [dnsext] What ELSE relies on UDP fragmentation?

From: Nicholas Weaver <[email-masked]>
Date: Tue, 18 Aug 2009 08:37:01 -0700

Can anyone identify a major application other than DNSSEC which uses  
UDP and assumes that packets greater than the Ethernet MTU can be sent  
and will be properly fragmented and reassembled on traffic traversing  
the Internet?

---

From: bert hubert <[email-masked]>
Date: Tue, 18 Aug 2009 18:11:58 +0200

On Tue, Aug 18, 2009 at 5:37 PM, Nicholas
Weaver<[email-masked]> wrote:
> Can anyone identify a major application other than DNSSEC which uses UDP and
> assumes that packets greater than the Ethernet MTU can be sent and will be
> properly fragmented and reassembled on traffic traversing the Internet?

"Fragmentation is like classful addressing -- an interesting early
 architectural error that shows how much experimentation was going
 on while IP was being designed."  -- Paul Vixie

Nothing else relies on fragmented UDP working as far as I know.
OpenVPN for example has no less than 7 configuration options to
prevent fragmentation, or to deal with it not working.

    Bert

---

From: Alex Bligh <[email-masked]>
Date: Tue, 18 Aug 2009 17:47:17 +0100

--On 18 August 2009 08:37:01 -0700 Nicholas Weaver 
<[email-masked]> wrote:

> Can anyone identify a major application other than DNSSEC which uses UDP
> and assumes that packets greater than the Ethernet MTU can be sent and
> will be properly fragmented and reassembled on traffic traversing the
> Internet?

NFS (at least v2) @rsize=4096,wsize=4096. Somehow I doubt this is
a major application though ...

-- 
Alex Bligh

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 18 Aug 2009 13:08:38 -0400

No hat.

On Tue, Aug 18, 2009 at 05:47:17PM +0100, Alex Bligh wrote:

> Somehow I doubt this is a major application though ...

I'm a little uncomfortable with the trend in this thread of trying to
discern the "major" applications as opposed to the "minor" ones, on
whom one gets to impose damage.  I suggest it's hard to make the
major/minor distinction in any clear way, and I also suggest that many
people who could in principle be relying on this feature are not
reading this list.  Therefore, it seems to me that if someone wants to
argue that some _N_ ought to be deprecated, the right answer is to
write the draft that deprecates that _N_, and circulate it very
widely.  Given the _N_ that I think is under consideration, this
mailing list is probably too restricted, although it's possible that
we're in the right area.

Let me emphasise that the above is a no-hat personal opinion.  I am
not moderating the conversation, insisting this discussion go
elsewhere, or making any determination of whether this discussion is
on-topic here.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 18 Aug 2009 10:42:04 -0700

> mailing list is probably too restricted, although it's possible that

Rather, the question is different:

Has reality effectively already deprecated _N_: EG, in this specific  
case, have all other applications effectively assumed UDP fragments  
over the Internet don't work?

So far, the only examples are:

DNSSEC: Effectively undeployed at scale, but significant suggestions  
that datagrams greater than the ethernet MTU will cause problems.

NSF v2: Already deprecated, and never intended for Internet usage in  
any case, but only within trusted networks.

We don't need a draft to deprecate something.  We just need to know if  
its already been deprecated underneath us.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 18 Aug 2009 13:48:55 -0400

> its already been deprecated underneath us.

Yes, I get this point.  But "We don't know about $APP," is by no means
the same as, "$APP does not exist."  Part of my point is that, if the
people happily building devices and applications out there were paying
attention to our list (or even if there were WG participants who
understood those builders as "important") then we wouldn't be in this
mess at all.  Especially in an old protocol like ours, the difference
between theory and practice is smallery in theory &c.

That isn't to say that a tentative conclusion, "_X_ is practically
speaking deprecated," is the wrong one.  But several
previously-interested real operators of ISP networks have made noisy
exits from the IETF (one of them used to be a Chair of this WG), and I
don't think we should be too sanguine on the basis of a survey of WG
participants.  That's what (and all) I'm saying.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: bert hubert <[email-masked]>
Date: Tue, 18 Aug 2009 20:04:51 +0200

> the same as, "$APP does not exist."  Part of my point is that, if the

We also know 8% (or was it 12%?) of Netalyzr users have issues with
fragmentation.

This sounds like it corroborates the 'we know of nothing else that
relies on it working end to end'.

Maybe DNSSEC won't be for everybody, but only for discerning users on
networks that are ok with (very) large packets?

     Bert

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Aug 2009 11:23:47 -0700

>Nothing else relies on fragmented UDP working as far as I know.

So, I guess you don't know about IKE, the keying protocol for IPsec. I could introduce you two. :-)

IKE runs over UDP, and always has. Some IKE messages are large due to them containing PKIX certificates and so on.

--Paul Hoffman, Director
--VPN Consortium

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 19 Aug 2009 03:17:49 +0800

> We also know 8% (or was it 12%?) of Netalyzr users have issues with

Is there a difference between IPv4 and IPv6? Or not enough IPv6  
traffic to be able to count?

    Patrik -- that have not seen this report, sorry...

---

From: bert hubert <[email-masked]>
Date: Tue, 18 Aug 2009 21:44:05 +0200

> IKE runs over UDP, and always has. Some IKE messages are large due to them containing PKIX certificates and so on.

This is indeed a poignant example. Especially since we find that IPSEC
vendors have decided to implement countermeasures, like the "IKE
Fragmentation" extension (which pre-fragments IKE messages over
multiple smaller UDP packets).

Or look at RFC3723 section 5.6 which discusses ways to deal with the
problems of UDP fragmentation preventing IKE operation. It suggests
reducing the size of certificates used.

In fact, it appears we could learn a lot from the travails of the
IPSEC community wrt fragmentation. I note that the Cisco
implementation decides that IKE messages above 576 bytes should
already be split out over multiple UDP packets...

    Bert

---

From: Paul Hoffman <[email-masked]>
Date: Tue, 18 Aug 2009 13:29:38 -0700

>multiple smaller UDP packets).

Some vendors have done this, of course. How does that relate to the original question:
At 8:37 AM -0700 8/18/09, Nicholas Weaver wrote:
>Can anyone identify a major application other than DNSSEC which uses UDP and assumes that packets greater than the Ethernet MTU can be sent and will be properly fragmented and reassembled on traffic traversing the Internet?

IKE uses UDP. IKE assumes that packets yadda yadda. Some vendors have created proprietary ways of preventing the problem of firewalls that block fragmented UDP.

>reducing the size of certificates used.

Yep.

>IPSEC community wrt fragmentation.

I'm not sure there is "a lot" to learn other than "the IPsec community has found that some firewalls block fragmented UDP; normal IKE sometimes produces fragmented UDP; therefore it is good to avoid producing fragmented UDP". The problem has not been so great that there is any work in the IPsec space to standardize ways to avoid fragmenting UDP. OTOH, DNSSEC might cause fragmented UDP more often than IKE does, given the extremely low usage of certificates in IKE, much less usage of certificates that are long enough to cause fragmentation.

--Paul Hoffman, Director
--VPN Consortium

---

From: bert hubert <[email-masked]>
Date: Tue, 18 Aug 2009 22:45:46 +0200

On Tue, Aug 18, 2009 at 10:29 PM, Paul Hoffman<[email-masked]> wrote:
> [...] given the extremely low usage of certificates in IKE, much less usage of certificates that are long enough to cause fragmentation.

So, getting back to Nicholas' question, IKE is not (generally) relying
on UDP fragmentation, except sometimes.

For those cases, vendors have felt the need to offer their customers
proprietary workarounds, presumably because there were problems,
enough of them to be noticeable.

I don't want to be nasty about this, but it does appear 'we' are
facing a very big problem with DNSSEC deployability as is, to the
point that I wonder if we'll need to move to ECDSA before confidently
being able to roll all of this out.

Don't forget that if people turn DNSSEC on, and it breaks, they won't
believe it next time implementors say it is 'ready for use'.

Nicholas, do you feel confident about the statistics you derive about
how many people can't see bigger packets?

     Bert

---

From: Paul Wouters <[email-masked]>
Date: Tue, 18 Aug 2009 16:45:54 -0400 (EDT)

> IKE runs over UDP, and always has. Some IKE messages are large due to them containing PKIX certificates and so on.

Have you seen that work? With openswan we don't really see that working at
all, and at times I have to tell people to go back to 1024bit RSA keys on
their X.509 certs if they need to be send with IKE.

So I'm not sure if this is a good example of "fragmenting udp works".

Paul

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 18 Aug 2009 17:12:03 -0400

No hat.  

> believe it next time implementors say it is 'ready for use'.

I think that's true.  The important question to ask, for deployment
purposes (and I suggest the answer to this belongs on another list),
is what "it breaks" means.

A

-- 
Andrew Sullivan
[email-masked]
Shinkuro, Inc.

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 18 Aug 2009 14:25:10 -0700

> how many people can't see bigger packets?

Somewhat confident but NOT completely.

Its enough to say "There is a potential problem here", the relative  
root causes of the problem (fragments vs DNS<=512B assumptions:  
fragments appear to be the big problem for recursive resolvers, and  
about 2/3rds for the end-host systems), but not exactly the scale of  
the problem: the error bars could be huge.

Simply because the Netalyzr run is an incredibly biased and self- 
selecting data-set.  Eg, how it got slashdotted caused ~10%+ of the  
tests to be from Comcast IPs!  We've identified some other biases as  
well, and who knows how many biases exist that we don't know about.

However, for those who are already doing monitoring of open recursive  
resolvers, the same basic probing methodology could be used (including  
the same authority we already have set up, so its really just a few  
more names into the query stream), and hopefully I can find someone  
who already has that infrastructure in place who'd be interested in  
adding the names to the queries, and possibly doing some other tests  
as well.

This would also be a biased data set (you only get open recursive  
resolvers and cases where someone has set up an open proxy to their  
recursive resolver), but by being a different set of biases, it should  
give us a better feel for what the real numbers are.

---

From: Paul Wouters <[email-masked]>
Date: Tue, 18 Aug 2009 17:56:07 -0400 (EDT)

> believe it next time implementors say it is 'ready for use'.

How many more decades are we help those people fix their routers and firewalls?

I mean, Cisco still says at www.cisco.com/web/about/security/intelligence/dns-bcp.html

 	DNS message size limitations

 	DNS amplification and reflection attacks are more effective when
 	leveraging large DNS messages than small DNS message sizes. The
 	message-length parameters submode command for policy-map type inspect
 	dns can be used to ensure that message sizes to not exceed a specified
 	size thus reducing the efficiency of these attacks.

 	This feature is available beginning with software release 7.2(1) for
 	Cisco ASA and Cisco PIX Firewalls. This feature is available beginning
 	with software release 3.1 for FWSM Firewalls. This function is enabled
 	by default with a limit of 512 bytes.

Seriously, if you're still setting "message-length maximum 512" in your network,
you deserve to keep all pieces.

A lot of people have already enabled and deployed DNSSEC. It will be hard for
anyone to say "dnssec is broken" instead of "my network is broken".

Paul

---

From: Doug Otis <[email-masked]>
Date: Tue, 18 Aug 2009 14:58:16 -0700

> is what "it breaks" means.

1) Reduction in IEEE CRC error detection reliably.

http://www.ece.cmu.edu/~koopman/networks/dsn02/dsn02_koopman.pdf

2) Fragmentation increases packet spoofing vulnerability.

3) Fragmentation increase resource exhaustion vulnerability.

4) Increased network amplification risks from none validated sources.

For example, current email authorization schemes might chain 
transactions via cached resource records and cause distributed 
recipients to modulate queries based upon email local-parts.

-Doug

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 19 Aug 2009 07:24:44 +0800

> is what "it breaks" means.

We have to be careful with the two for me different questions:

- Do fragmented UDP packets drop in a rate that is so high that we can  
not rely on it working for any existing or future protocol in the IETF

- UDP packet fragmentation issues create problems for DNSSEC in real  
life

The second of course relies on the fragmentation actually happen for  
DNSSEC signed responses, and as I think most fragmentation issues  
happens in soho firewalls, that DNSSEC signed responses do not  
traverse, it is not much of a problem.

Or let me put this differently:

In Sweden where we do have some signed domains, we have not heard of  
this problem.

What we do see problems with have to do with key rollover, registry/ 
registrar problems etc. But not fragmentation of UDP packets.

My guess is that this might be because verification is NOT happening  
on the customer side of the SOHO routers that are said to create  
problems with UDP fragmentation, which implies we still do not know  
what is happening, and whether the statement actually is true, "do  
fragmentation of UDP packets create problems for DNSSEC deployment".

    paf

---

From: Patrik Fältström <[email-masked]>
Date: Wed, 19 Aug 2009 07:33:19 +0800

> you deserve to keep all pieces.

Note that this is a function in a firewall, and this has nothing to do  
with IP fragmentation as this is an inspection of the DNS payload size.

That said, I agree with you. If one have firewalls set to ensure that  
DNS payload MUST be smaller than 512 bytes, and still try to run  
DNSSEC (which of course will fail), then one should deserve to loose.

Note further than the feature you describe is a bit more delicate than  
the text above describe. The feature also recognize the ENDS0  
announcement of the response buffer size, so if you send out a request  
with EDNS0 size limitation, the firewall inspects the size of the  
response that it is not larger than what you announced. So the text  
you sent is regarding non-EDNS0 tagged packets. But in reality the  
algorithm used is even more complicated, so people that want to know  
how this inspection works should read the full documentation.

    Patrik

---

From: Mark Andrews <[email-masked]>
Date: Wed, 19 Aug 2009 11:33:33 +1000

In message <[email-masked]>, =?ISO-8859-1?Q?Pat
rik_F=E4ltstr=F6m?= writes:
> This is an OpenPGP/MIME signed message (RFC 2440 and 3156)

	Wouldn't make more sense to fix that documentation to say
	something like:

	by default with a limit of 512 bytes for plain DNS.  EDNS
	request packets are inspected to see what the EDNS UDP size
	is and the responses are examined to ensure that they conform.

	We would then have a club to beat up other firewall vendors
	with by saying "CISCO can do this so you should be able to
	do this as well" and point them at your page.

	We would also have a page we can point to which gives the
	minimum version CISCO customers need to be running which
	is compatible with DNSSEC.

	Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Alex Bligh <[email-masked]>
Date: Wed, 19 Aug 2009 07:28:06 +0100

> 	is and the responses are examined to ensure that they conform.

That would be lovely, and might be an accurate description of
what it does now. However, it does not appear to be an accurate
description of what the feature did when first introduced, judging
by a lot of postings to the net by people who claim it broke
EDNS queries beyond 512. Of course, a page saying Cisco fixed
it would be even better.

-- 
Alex Bligh

---

From: Alex Bligh <[email-masked]>
Date: Wed, 19 Aug 2009 07:47:17 +0100

--On 18 August 2009 14:25:10 -0700 Nicholas Weaver 
<[email-masked]> wrote:

> Its enough to say "There is a potential problem here", the relative root
> causes of the problem (fragments vs DNS<=512B assumptions: fragments
> appear to be the big problem for recursive resolvers, and about 2/3rds
> for the end-host systems), but not exactly the scale of the problem: the
> error bars could be huge.
>
> Simply because the Netalyzr run is an incredibly biased and
> self-selecting data-set.  Eg, how it got slashdotted caused ~10%+ of the

OK, so I ran netalyzer from my OS-X laptop. It is running with a separate
DNS server discovered through DHCP (modern-ish BIND), behind a cruftily
configured Cisco router as a firewall & NAT. The BIND server (Linux
box) receives no special treatment on the firewall.

This is one of your data points that is unable to receive fragments.
However, I have no DNS problems I know about (certainly not receiving
NXDOMAIN from .se).

It claims that "Large DNS requests are blocked.  Grr" (in the log). I
don't believe this is actually the case. Is there a simple CLI utility
to test this?

It also claims:

  052.539 test-22| Now checking for a bogus (non-DNS) packet on port 53
  052.540 test-22| UDP socket at 192.168.100.80:53561
  052.548 test-22| Got exception java.net.SocketException: Network is 
unreachable on UDP test
  052.549 test-22| Bogus ping packet not received by the applet
  052.549 test-22| now checking again with a real DNS packet
  052.550 test-22| UDP socket at 192.168.100.80:53562
  052.648 test-22| Got datagram of 212 bytes.
  052.650 test-22| Legitimate DNS packet succeeded.

And hence malformed DNS packets are being dropped. I am about 99% sure
that it's not the path out doing this. It's either Java or some internal
OS-X firewall prevent user applications from constructing their own
invalid DNS packets.

My concern is that Netalyzer may be reporting problems where there
are in fact none.

-- 
Alex Bligh

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 19 Aug 2009 05:38:07 -0700

On Aug 18, 2009, at 11:47 PM, Alex Bligh wrote:

>
>
> --On 18 August 2009 14:25:10 -0700 Nicholas Weaver <[email-masked] 
> > wrote:
>
>> root

Does the RESOLVER get large responses OK?  Or is it just your host?   
Remember, we test both of those separately.

> don't believe this is actually the case. Is there a simple CLI utility

dig +norecurse +dnssec edns_large.netalyzr.icsi.berkeley.edu  
@n1.netalyzr.icsi.berkeley.edu

dig +norecurse +dnssec edns_medium.netalyzr.icsi.berkeley.edu  
@n1.netalyzr.icsi.berkeley.edu

dig +norecurse +dnssec edns_small.netalyzr.icsi.berkeley.edu  
@n1.netalyzr.icsi.berkeley.edu

are the manually constructed requests: its a direct UDP raw packet  
equivelent to the dig request (in fact, thats how I generated it, I  
just used dig and captured the packet, easier than finding a Java DNS  
lib I liked).

The responses should have a large number of irrelevant CNAMEs to pad  
out the packet.

> OS-X firewall prevent user applications from constructing their own

The default Java and OS-X will, just fine, send that bogus packet,  
under both firefox and safari.

Could you send me the link to the results page/transcript so I can see  
if there is a different JVM version?

What other security software are you using?

Also, many firewalls (you mentioned this as being behind a crufty  
Cisco, is it effectively local network?) WILL block the bogus request,  
and I suspect if it actually gives an ICMP back rather than just  
silently dropping, Java will get that exception.

Can you try a simple dumb raw UDP request to that port and see what  
you get back?

---

From: Alex Bligh <[email-masked]>
Date: Wed, 19 Aug 2009 21:03:34 +0100

--On 19 August 2009 07:47:17 +0100 Alex Bligh <[email-masked]> wrote:

> My concern is that Netalyzr may be reporting problems where there
> are in fact none.

This turns out to be that Cisco NAT silently does even more bizarre things
with DNS packets than one might have thought. Netalyzr is working just
fine.

-- 
Alex Bligh