Subject: [v6ops] OSX 10.8 poor performance for IPv6-only!

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Tue, 29 Jan 2013 17:18:24 +0000

A weird behavior of OSX 10.8.2 wrt DNS A and AAAA lookup (with IPv6 GUA,
but no IPv4 public/private address) is intermittently causing very poor
browser internet experience on the MacBook.

MBP nicely issues both DNS A and AAAA queries over IPv6 transport for a
given website (say, www.cnn.com) almost at the same time (e.g. opens 2 UDP
sockets), and receives both A and AAAA responses from the DNS server.
However, it may close both UDP sockets upon receiving the first DNS
response (whether A or AAAA), thereby ignoring the 2nd DNS response
(whether A or AAAA).

Suffice to say that MBP issues an ICMPv6 Destination/port unreachable for
the 2nd DNS response (see the attached 2 screen wireshark screenshots).

This is quite bad, since if an AAAA record was carried in the ignored DNS
message, then no TCP connection would follow => forget happy eyeballs and
recall poor internet experience. :(

Has anybody else run into this or noticed it?

Cheers,
Rajiv

---

From: Hiroaki Hazeyama <[email-masked]>
Date: Wed, 30 Jan 2013 02:23:25 +0900

we reported it in 4.2.2.2 of 
http://tools.ietf.org/html/draft-hazeyama-widecamp-ipv6-only-experience-02

----
hazeyama

(2013/01/30 2:18), Rajiv Asati (rajiva) wrote:
> _______________________________________________
> v6ops mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/v6ops
>

---

From: David Farmer <[email-masked]>
Date: Tue, 29 Jan 2013 12:15:29 -0600

I believe this is the "ARP for everything" behavior defined in section 2.6.2 of RFC 3927 which enables a device with a link local address to communicate with devices on the local link that are using IPv4 GUA.  You will find a default route to the local interface, not a router, that enables this behavior.

I learned of this from some IPv6 only experiments Ron Broersma reported on at a recent Internet2 IPv6 working group meeting.

I believe full implementation of section 3.2 of RFC 6724 would depreference the IPv4 link-local address in favor of a IPv6 GUA.  I've been thing about possible operational work-arounds, best current thought is a device set up to proxy-ARP for everything and return an immediate destination unreachable, but haven't tried it yet.  

However, obviously the best solution is for hosts to never prefer a IPv4 link-local address over an IPv6 GUA when communicating with a GUA address.

-- 
===============================================
David Farmer               Email:[email-masked]
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota    
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

On Jan 29, 2013, at 11:18, "Rajiv Asati (rajiva)" <[email-masked]> wrote:

> <Screen Shot 2013-01-29 at 11.38.35 AM.png>
> <Screen Shot 2013-01-29 at 11.40.04 AM.png>

---

From: Rémi Després <[email-masked]>
Date: Tue, 29 Jan 2013 19:29:13 +0100

Hi, Raji,

I didn't make a detailed study like you did to understand what happens, but I observed that, while I had been reaching Google in IPv6 for several years now, I tend to be back in IPv4 now, a real pity.

Regards,
RD

Le 2013-01-29 à 18:18, Rajiv Asati (rajiva) <[email-masked]> a écrit :

> <Screen Shot 2013-01-29 at 11.38.35 AM.png><Screen Shot 2013-01-29 at 11.40.04 AM.png>_______________________________________________
> v6ops mailing list
> [email-masked]

---

From: Marc Blanchet <[email-masked]>
Date: Tue, 29 Jan 2013 13:30:52 -0500

> I didn't make a detailed study like you did to understand what happens, but I observed that, while I had been reaching Google in IPv6 for several years now, I tend to be back in IPv4 now, a real pi…

does not seem relevant, since Rajiv wrote: "but no IPv4 public/private address"

Marc.

---

From: Rémi Després <[email-masked]>
Date: Tue, 29 Jan 2013 19:40:29 +0100

> does not seem relevant, since Rajiv wrote: "but no IPv4 public/private address"

Not sure I have the time to look at more details.
FWIW, my IPv4 address is 192.168.0.21 (private behind a NAT).

I only know that my use of IPv6 has been deteriorated, without a change of my IPv6 and IPv4 addresses.

RD

---

From: Owen DeLong <[email-masked]>
Date: Tue, 29 Jan 2013 13:16:41 -0800

> I believe this is the "ARP for everything" behavior defined in section 2.6.2 of RFC 3927 which enables a device with a link local address to communicate with devices on the local link that are usin…

Nope... this is more detailed and more insidious. If it is not waiting for the second DNS response and the AAAA record happens to not be the first answer, bad stuff happens regardless of whether IPv4 is enabled on the interface (arp for everything problem) or not.

This is an actual bug on Apple's part if they are prematurely closing the DNS connection for the second request as he describes.

Remember, getaddrinfo() actually makes two requests to the DNS server. One for the A and one for the AAAA. The responses arrive on separate channels as well.

This sounds like a bug in Apple's implementation of getaddrinfo() or its underlying libraries.

> I believe full implementation of section 3.2 of RFC 6724 would depreference the IPv4 link-local address in favor of a IPv6 GUA.  I've been thing about possible operational work-arounds, best curren…

That doesn't help if you never get the AAAA record.

Owen

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Wed, 30 Jan 2013 00:27:22 +0000

Owen, you are spot on. 

I can't think of anything, but a possible bug in OSX 10.8.2. Really a weird one. I wonder whether this was meant to make the happy-eyeballing implementation better. ;-)

My IPv6-only usage has been pathetically slow on MBP, least to say. I iPhone has been fine though.

Cheers,
Rajiv

Sent from my Phone

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 30 Jan 2013 09:34:31 +0900

Why is it requesting an A record even if it has no IPv4 address? Isn't that
what AI_ADDRCONFIG is for?

Is it possible that the browser is not passing in AI_ADDRCONFIG? Do
different browsers behave in different ways?

On Wed, Jan 30, 2013 at 9:27 AM, Rajiv Asati (rajiva) <[email-masked]>wrote:

> Owen, you are spot on.
>
> I can't think of anything, but a possible bug in OSX 10.8.2. Really a
> weird one. I wonder whether this was meant to make the happy-eyeballing
> implementation better. ;-)
>
> >> I believe this is the "ARP for everything" behavior defined in section
> 2.6.2 of RFC 3927 which enables a device with a link local address to
> communicate with devices on the local link that are using IPv4 GUA.  You
> will find a default route to the local interface, not a router, that
> enables this behavior.
> >
> > Nope... this is more detailed and more insidious. If it is not waiting
> for the second DNS response and the AAAA record happens to not be the first
> answer, bad stuff happens regardless of whether IPv4 is enabled on the
> interface (arp for everything problem) or not.
> >
> > This is an actual bug on Apple's part if they are prematurely closing
> the DNS connection for the second request as he describes.
> >
> > Remember, getaddrinfo() actually makes two requests to the DNS server.
> One for the A and one for the AAAA. The responses arrive on separate
> channels as well.
> >
> >> I learned of this from some IPv6 only experiments Ron Broersma reported
> on at a recent Internet2 IPv6 working group meeting.
> >>
> >> I believe full implementation of section 3.2 of RFC 6724 would
> depreference the IPv4 link-local address in favor of a IPv6 GUA.  I've been
> thing about possible operational work-arounds, best current thought is a
> device set up to proxy-ARP for everything and return an immediate
> destination unreachable, but haven't tried it yet.
> >
> >> However, obviously the best solution is for hosts to never prefer a
> IPv4 link-local address over an IPv6 GUA when communicating with a GUA
> address.
> >>
> >> --

---

From: Owen DeLong <[email-masked]>
Date: Tue, 29 Jan 2013 16:38:17 -0800

Not so strange... I suspect the logic went something like this...

Happy Eyeballs is all about returning a usable connection as fast as possible.

Most testing likely done in dual-stack and/or IPv4-only environments.

DNS resolver library sends out A and AAAA requests.

It's not at all hard to imagine how a developer could make the mistake of either of the following logics:

while (!timeout)
{
  if(v4_answer_received && v6_answer_received) return(concatenate(v4_answer, v6_answer));
  if(v4_answer_received) return(v4_answer);
}

or

while (!timeout)
{
  if(v4_answer_received && v6_answer_received) return(concatenate(v4_answer, v6_answer));
  if(v4_answer_received) return(v4_answer);
  if(v6_answer_received) return(v6_answer);
}

In the first case, it would break on an IPv6 only network, but work in an IPv4-only environment.
In the second case, it would break in certain IPv4-only environments, too.

Obviously, the real bug is going to be more complex and integral, as Happy Eyeballs is
a combination of resolver and connection initiation, likely involving a threaded approach,
but I wanted to provide a simple illustration of the likely logic trap. I'll leave the detailed
debugging to Apple's engineers.

Owen

---

From: Lorenzo Colitti <[email-masked]>
Date: Wed, 30 Jan 2013 09:40:50 +0900

On Wed, Jan 30, 2013 at 9:38 AM, Owen DeLong <[email-masked]> wrote:

> I'll leave the detailed debugging to Apple's engineers.
>

Good luck with that. :)

---

From: Mark Andrews <[email-masked]>
Date: Wed, 30 Jan 2013 12:06:31 +1100

Part of the problem is there is this myth that AAAA lookups fail a
lot.  They don't.  There are a few bad (drop AAAA queries) /
misconfigured (return the wrong negative answer) load balancers.
If you are talking to something other than a load balancer you get
valid answers.  When you code for this you get all sorts of bad
behaviour.

If you can't configure your nameserver to respond correctly to AAAA
queries the customet SHOULD get bad performance.  How else does
feedback get applied.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Owen DeLong <[email-masked]>
Date: Tue, 29 Jan 2013 19:04:59 -0800

Providing feedback to the end-user about a DNS server they don't even
know they are using in a way that does not allow them to easily identify the
DNS server in question, let alone who is responsible for it or how to report
the problem isn't useful.

Feedback that reaches the person responsible for fixing the problem is
like feedback heard by the sound-man in a concert hall. The sound man
will take action to resolve the issue.

Feedback that doesn't is just noise that makes everyone in the hall unhappy.
However, regardless of how unhappy they are, they don't have any real
ability to fix the problem.

Owen

---

From: Mark Andrews <[email-masked]>
Date: Wed, 30 Jan 2013 15:41:30 +1100

In message <[email-masked]>, Owen DeLong write
> the problem isn't useful.

How does a user deal with any remote problem?  They contact their
ISP.  The ISP should have enough clue to contact the DNS server
administrator.  IPv6 is being deployed wide enough that even IPv4
only sites are starting to be aware of it.

Note this is also a problem that will correct itself over time as
sites with broken load balancers enable IPv6.

I just for interest I dumped the nameserver's cache and looked up
the AAAA for every A record in the cache.  I had one zone with
broken AAAA lookups and one lookup failing due to DNSSEC validation
failures.

The servers for cedexis.net return A records in response to AAAA
lookups.

		flipa.cedexis.net.
		flipd.cedexis.net.
		flipg.cedexis.net.

Mark

---

From: Philipp Kern <[email-masked]>
Date: Wed, 30 Jan 2013 10:26:43 +0100

Mark,

> misconfigured (return the wrong negative answer) load balancers.

and CPEs. (Which meant switching to Google DNS for the Linux clients in
that network.) Feedback's useless, it doesn't get forwarded to the right
places within the ISP and/or the ISP doesn't care and/or the CPE vendor
doesn't put out a fix.

The broken (not misconfigured) load balancers were dropped from their
recursors by another ISP when I mentioned it to them. That's on the
network level and somehow easier to solve.

Kind regards
Philipp Kern

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Thu, 31 Jan 2013 18:02:34 +0000

Lorenzo,

The host does have an IPv4 LLA, so AI_ADDRCONFIG flag would cause A query
to happen.

Cheers,
Rajiv

-----Original Message-----
From: Lorenzo Colitti <[email-masked]>
Date: Tuesday, January 29, 2013 7:34 PM
To: Rajiv Asati <[email-masked]>
Cc: Owen DeLong <[email-masked]>, "[email-masked]" <[email-masked]>
Subject: Re: [v6ops] OSX 10.8 poor performance for IPv6-only!

>Why is it requesting an A record even if it has no IPv4 address? Isn't
>that what AI_ADDRCONFIG is for?
>
>
>On Wed, Jan 30, 2013 at 9:27 AM, Rajiv Asati (rajiva)
><[email-masked]> wrote:
>
>Owen, you are spot on.
>
>>>communicate with devices on the local link that are using IPv4 GUA.
>>>You will find a default route to the local interface,
> not a router, that enables this behavior.
>>
>>for the second DNS response and the AAAA record happens to not be the
>>first answer, bad stuff happens regardless of whether IPv4 is enabled on
>>the interface (arp for everything problem)
> or not.
>>
>>> I learned of this from some IPv6 only experiments Ron Broersma
>>>reported on at a recent Internet2 IPv6 working group meeting.
>>>
>>>depreference the IPv4 link-local address in favor of a IPv6 GUA.  I've
>>>been thing about possible operational work-arounds, best current
>>>thought is a device set up to proxy-ARP for everything
> and return an immediate destination unreachable, but haven't tried it
>yet.
>>
>>><mailto:[email-masked]>
>>> 2218 University Ave SE        Phone:
>612-626-0815 <tel:612-626-0815>
>>> Minneapolis, MN 55414-3029   Cell:
>612-812-9952 <tel:612-812-9952>
>>>> given website (say, www.cnn.com <http://www.cnn.com>) almost at the
>>>>same time (e.g. opens 2 UDP

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Thu, 31 Jan 2013 18:05:45 +0000

Owen,

That's a fair guesswork. Hopefully, someone from OSX could chime in.

I will try to disable IPv4 altogether and check if I see a different
behavior.

Cheers,
Rajiv

-----Original Message-----
From: Owen DeLong <[email-masked]>
Date: Tuesday, January 29, 2013 7:38 PM
To: Rajiv Asati <[email-masked]>
Cc: David Farmer <[email-masked]>, "[email-masked]" <[email-masked]>, Ron
Broersma <[email-masked]>
Subject: Re: [v6ops] OSX 10.8 poor performance for IPv6-only!

>  if(v4_answer_received && v6_answer_received)
>return(concatenate(v4_answer, v6_answer));
>  if(v4_answer_received && v6_answer_received)
>return(concatenate(v4_answer, v6_answer));
>>>> I believe this is the "ARP for everything" behavior defined in
>>>>section 2.6.2 of RFC 3927 which enables a device with a link local
>>>>address to communicate with devices on the local link that are using
>>>>IPv4 GUA.  You will find a default route to the local interface, not a
>>>>router, that enables this behavior.
>>> 
>>>first answer, bad stuff happens regardless of whether IPv4 is enabled
>>>on the interface (arp for everything problem) or not.
>>> 
>>>>thought is a device set up to proxy-ARP for everything and return an
>>>>immediate destination unreachable, but haven't tried it yet.
>>>

---

From: "Rajiv Asati (rajiva)" <[email-masked]>
Date: Thu, 31 Jan 2013 18:06:29 +0000

Hi Mark,

HmmmŠ is it valid to return AAAA record for A query?

Cheers,
Rajiv

---

From: Mark Andrews <[email-masked]>
Date: Fri, 01 Feb 2013 07:37:21 +1100

In message <[email-masked]>, "R
ajiv Asati (rajiva)" writes:
> Hi Mark,
> 
> Hmmm=8A is it valid to return AAAA record for A query?

No.

The owner of the servers is looking into the issue.  Got a polite thank
you for informing us, we will look into it.  I don't expect a instant fix
as they will need to talk with their name server vendor.

Mark